An Analysis of Dropout for Matrix Factorization

by   Jacopo Cavazza, et al.

Dropout is a simple yet effective algorithm for regularizing neural networks by randomly dropping out units through Bernoulli multiplicative noise, and for some restricted problem classes, such as linear or logistic regression, several theoretical studies have demonstrated the equivalence between dropout and a fully deterministic optimization problem with data-dependent Tikhonov regularization. This work presents a theoretical analysis of dropout for matrix factorization, where Bernoulli random variables are used to drop a factor, thereby attempting to control the size of the factorization. While recent work has demonstrated the empirical effectiveness of dropout for matrix factorization, a theoretical understanding of the regularization properties of dropout in this context remains elusive. This work demonstrates the equivalence between dropout and a fully deterministic model for matrix factorization in which the factors are regularized by the sum of the product of the norms of the columns. While the resulting regularizer is closely related to a variational form of the nuclear norm, suggesting that dropout may limit the size of the factorization, we show that it is possible to trivially lower the objective value by doubling the size of the factorization. We show that this problem is caused by the use of a fixed dropout rate, which motivates the use of a rate that increases with the size of the factorization. Synthetic experiments validate our theoretical findings.


page 1

page 2

page 3

page 4


Dropout as a Low-Rank Regularizer for Matrix Factorization

Regularization for matrix factorization (MF) and approximation problems ...

On Dropout and Nuclear Norm Regularization

We give a formal and complete characterization of the explicit regulariz...

Asymptotic convergence rate of Dropout on shallow linear neural networks

We analyze the convergence rate of gradient flows on objective functions...

On the Inductive Bias of Dropout

Dropout is a simple but effective technique for learning in neural netwo...

Dropout: Explicit Forms and Capacity Control

We investigate the capacity control provided by dropout in various machi...

Fast and Exact Matrix Factorization Updates for Nonlinear Programming

LU and Cholesky matrix factorization algorithms are core subroutines use...

Robust Volume Minimization-Based Matrix Factorization for Remote Sensing and Document Clustering

This paper considers volume minimization (VolMin)-based structured matri...

Please sign up or login with your details

Forgot password? Click here to reset