Transportation analysis of denoising autoencoders: a novel method for analyzing deep neural networks

by   Sho Sonoda, et al.

The feature map obtained from the denoising autoencoder (DAE) is investigated by determining transportation dynamics of the DAE, which is a cornerstone for deep learning. Despite the rapid development in its application, deep neural networks remain analytically unexplained, because the feature maps are nested and parameters are not faithful. In this paper, we address the problem of the formulation of nested complex of parameters by regarding the feature map as a transport map. Even when a feature map has different dimensions between input and output, we can regard it as a transportation map by considering that both the input and output spaces are embedded in a common high-dimensional space. In addition, the trajectory is a geometric object and thus, is independent of parameterization. In this manner, transportation can be regarded as a universal character of deep neural networks. By determining and analyzing the transportation dynamics, we can understand the behavior of a deep neural network. In this paper, we investigate a fundamental case of deep neural networks: the DAE. We derive the transport map of the DAE, and reveal that the infinitely deep DAE transports mass to decrease a certain quantity, such as entropy, of the data distribution. These results though analytically simple, shed light on the correspondence between deep neural networks and the Wasserstein gradient flows.


page 1

page 2

page 3

page 4


Learning High Dimensional Wasserstein Geodesics

We propose a new formulation and learning strategy for computing the Was...

Generative Modeling with Optimal Transport Maps

With the discovery of Wasserstein GANs, Optimal Transport (OT) has becom...

Beyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks

Deep neural networks often consist of a great number of trainable parame...

Transport map unadjusted Langevin algorithms

Langevin dynamics are widely used in sampling high-dimensional, non-Gaus...

Survey of Expressivity in Deep Neural Networks

We survey results on neural network expressivity described in "On the Ex...

Stacked Autoencoder Based Deep Random Vector Functional Link Neural Network for Classification

Extreme learning machine (ELM), which can be viewed as a variant of Rand...

On the Learning Dynamics of Deep Neural Networks

While a lot of progress has been made in recent years, the dynamics of l...

Please sign up or login with your details

Forgot password? Click here to reset