The Local Dimension of Deep Manifold

by   Mengxiao Zhang, et al.

Based on our observation that there exists a dramatic drop for the singular values of the fully connected layers or a single feature map of the convolutional layer, and that the dimension of the concatenated feature vector almost equals the summation of the dimension on each feature map, we propose a singular value decomposition (SVD) based approach to estimate the dimension of the deep manifolds for a typical convolutional neural network VGG19. We choose three categories from the ImageNet, namely Persian Cat, Container Ship and Volcano, and determine the local dimension of the deep manifolds of the deep layers through the tangent space of a target image. Through several augmentation methods, we found that the Gaussian noise method is closer to the intrinsic dimension, as by adding random noise to an image we are moving in an arbitrary dimension, and when the rank of the feature matrix of the augmented images does not increase we are very close to the local dimension of the manifold. We also estimate the dimension of the deep manifold based on the tangent space for each of the maxpooling layers. Our results show that the dimensions of different categories are close to each other and decline quickly along the convolutional layers and fully connected layers. Furthermore, we show that the dimensions decline quickly inside the Conv5 layer. Our work provides new insights for the intrinsic structure of deep neural networks and helps unveiling the inner organization of the black box of deep neural networks.


page 6

page 11


The SVD of Convolutional Weights: A CNN Interpretability Framework

Deep neural networks used for image classification often use convolution...

Deep Learning Weight Pruning with RMT-SVD: Increasing Accuracy and Reducing Overfitting

In this work, we present some applications of random matrix theory for t...

A Multi-modal Deep Neural Network approach to Bird-song identification

We present a multi-modal Deep Neural Network (DNN) approach for bird son...

The Singular Values of Convolutional Layers

We characterize the singular values of the linear transformation associa...

Curvature-based Comparison of Two Neural Networks

In this paper we show the similarities and differences of two deep neura...

Singular Value Representation: A New Graph Perspective On Neural Networks

We introduce the Singular Value Representation (SVR), a new method to re...

Data Representations' Study of Latent Image Manifolds

Deep neural networks have been demonstrated to achieve phenomenal succes...

Please sign up or login with your details

Forgot password? Click here to reset