How Wide Convolutional Neural Networks Learn Hierarchical Tasks

by   Francesco Cagnetta, et al.

Despite their success, understanding how convolutional neural networks (CNNs) can efficiently learn high-dimensional functions remains a fundamental challenge. A popular belief is that these models harness the compositional and hierarchical structure of natural data such as images. Yet, we lack a quantitative understanding of how such structure affects performances, e.g. the rate of decay of the generalisation error with the number of training samples. In this paper we study deep CNNs in the kernel regime: i) we show that the spectrum of the corresponding kernel and its asymptotics inherit the hierarchical structure of the network; ii) we use generalisation bounds to prove that deep CNNs adapt to the spatial scale of the target function; iii) we illustrate this result by computing the rate of decay of the error in a teacher-student setting, where a deep CNN is trained on the output of another deep CNN with randomly-initialised parameters. We find that if the teacher function depends on certain low-dimensional subsets of the input variables, then the rate is controlled by the effective dimensionality of these subsets. Conversely, if the teacher function depends on the full set of input variables, then the error rate is inversely proportional to the input dimension. Interestingly, this implies that despite their hierarchical structure, the functions generated by deep CNNs are too rich to be efficiently learnable in high dimension.


page 1

page 2

page 3

page 4


How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model

Learning generic high-dimensional tasks is notably hard, as it requires ...

Deep Quantile Regression: Mitigating the Curse of Dimensionality Through Composition

This paper considers the problem of nonparametric quantile regression un...

Locality defeats the curse of dimensionality in convolutional teacher-student scenarios

Convolutional neural networks perform a local and translationally-invari...

Can CNNs Construct Highly Accurate Model Efficiently with Limited Training Samples?

It is well known that metamodel or surrogate modeling techniques have be...

Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks

Convolutional residual neural networks (ConvResNets), though overparamet...

How isotropic kernels learn simple invariants

We investigate how the training curve of isotropic kernel methods depend...

Please sign up or login with your details

Forgot password? Click here to reset