Stronger convergence results for deep residual networks: network width scales linearly with training data size

• Main Author: Gulcu, T.C (Author)
• Format: Article
• Language: English
• Published: Oxford University Press 2022
• Subjects: activation function; deep network optimization; deep residual networks; neural tangent kernel
• Online Access: View Fulltext in Publisher

 Deep neural networks are highly expressive machine learning models with the ability to interpolate arbitrary datasets. Deep nets are typically optimized via first-order methods, and the optimization process crucially depends on the characteristics of the network as well as the dataset. This work sheds light on the relation between the network size and the properties of the dataset with an emphasis on deep residual networks (ResNets). Our contribution is that if the network Jacobian is full rank, gradient descent for the quadratic loss and smooth activation converges to the global minima even if the network width    of the ResNet scales linearly with the sample size    and logarithmically with the network depth     Consequently, our work is able to provide a theoretical guarantee for the convergence of deep neural networks in the    =\varOmega (n,\log H)   regime. © 2020 The Author(s) 2019. Published by Oxford University Press on behalf of the Institute of Mathematics and its Applications. All rights reserved.