On the Optimization Dynamics of Wide Hypernetworks

by   Etai Littwin, et al.

Recent results in the theoretical study of deep learning have shown that the optimization dynamics of wide neural networks exhibit a surprisingly simple behaviour. In this work, we study the optimization dynamics of hypernetworks, which are architectures in which a learned meta-network produces the weights of a task-specific primary network. Hypernetworks have been demonstrated repeatedly to obtain state of the art results. However, their theoretical understanding is still lacking. As can be expected, the optimization process of multiplicative models is much more complicated than optimizing standard ReLU networks. It is shown that for an infinitely wide neural network with a gating layer the cost function cannot be accurately approximated by it first order Taylor approximation. Specifically, for a fixed sized primary network of depth H, the first H terms of the Taylor approximation of the cost function are non-zero, even when the meta-network is infinitely wide. However, for an infinitely wide meta and primary networks, the learning dynamics is determined by a linear model obtained from the first-order Taylor expansion of the network around its initial parameters and the kernel of this process is given by the Hadamard product of the kernels induced by the meta and primary networks. As part of our study, we partially solve an open problem suggested by Dyer Gur-Ari (2020) and show that the convergence rate of the r order term of the Taylor expansion of the cost function, along the optimization trajectories of SGD is n^1-r, where n is the width of the learned neural network, improving upon the bound suggested by the conjecture of Dyer Gur-Ari, while matching their empirical observations. This result extends the recent n^-1 second order upper bound of Hanin Nica (2020).


page 1

page 2

page 3

page 4


Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

A longstanding goal in deep learning research has been to precisely char...

Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width

We propose Taylorized training as an initiative towards better understan...

Function approximation by deep neural networks with parameters {0,±1/2, ± 1, 2}

In this paper it is shown that C_β-smooth functions can be approximated ...

Infinite-width limit of deep linear neural networks

This paper studies the infinite-width limit of deep linear neural networ...

Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networks

The training process of ReLU neural networks often exhibits complicated ...

Connecting NTK and NNGP: A Unified Theoretical Framework for Neural Network Learning Dynamics in the Kernel Regime

Artificial neural networks have revolutionized machine learning in recen...

Look Wider to Match Image Patches with Convolutional Neural Networks

When a human matches two images, the viewer has a natural tendency to vi...

Please sign up or login with your details

Forgot password? Click here to reset