Weight Expansion: A New Perspective on Dropout and Generalization

by   Gaojie Jin, et al.

While dropout is known to be a successful regularization technique, insights into the mechanisms that lead to this success are still lacking. We introduce the concept of weight expansion, an increase in the signed volume of a parallelotope spanned by the column or row vectors of the weight covariance matrix, and show that weight expansion is an effective means of increasing the generalization in a PAC-Bayesian setting. We provide a theoretical argument that dropout leads to weight expansion and extensive empirical support for the correlation between dropout and weight expansion. To support our hypothesis that weight expansion can be regarded as an indicator of the enhanced generalization capability endowed by dropout, and not just as a mere by-product, we have studied other methods that achieve weight expansion (resp.contraction), and found that they generally lead to an increased (resp.decreased) generalization ability. This suggests that dropout is an attractive regularizer, because it is a computationally cheap method for obtaining weight expansion. This insight justifies the role of dropout as a regularizer, while paving the way for identifying regularizers that promise improved generalization through weight expansion.


page 1

page 2

page 3

page 4


A Bayesian encourages dropout

Dropout is one of the key techniques to prevent the learning from overfi...

On Dropout, Overfitting, and Interaction Effects in Deep Neural Networks

We examine Dropout through the perspective of interactions: learned effe...

Dropout: Explicit Forms and Capacity Control

We investigate the capacity control provided by dropout in various machi...

On the Inductive Bias of Dropout

Dropout is a simple but effective technique for learning in neural netwo...

Neural Network Regularization via Robust Weight Factorization

Regularization is essential when training large neural networks. As deep...

Dropout as a Low-Rank Regularizer for Matrix Factorization

Regularization for matrix factorization (MF) and approximation problems ...

Please sign up or login with your details

Forgot password? Click here to reset