A Sparse Coding Interpretation of Neural Networks and Theoretical Implications

08/14/2021
by   Joshua Bowren, et al.
0

Neural networks, specifically deep convolutional neural networks, have achieved unprecedented performance in various computer vision tasks, but the rationale for the computations and structures of successful neural networks is not fully understood. Theories abound for the aptitude of convolutional neural networks for image classification, but less is understood about why such models would be capable of complex visual tasks such as inference and anomaly identification. Here, we propose a sparse coding interpretation of neural networks that have ReLU activation and of convolutional neural networks in particular. In sparse coding, when the model's basis functions are assumed to be orthogonal, the optimal coefficients are given by the soft-threshold function of the basis functions projected onto the input image. In a non-negative variant of sparse coding, the soft-threshold function becomes a ReLU. Here, we derive these solutions via sparse coding with orthogonal-assumed basis functions, then we derive the convolutional neural network forward transformation from a modified non-negative orthogonal sparse coding model with an exponential prior parameter for each sparse coding coefficient. Next, we derive a complete convolutional neural network without normalization and pooling by adding logistic regression to a hierarchical sparse coding model. Finally we motivate potentially more robust forward transformations by maintaining sparse priors in convolutional neural networks as well performing a stronger nonlinear transformation.

READ FULL TEXT

page 10

page 11

research
08/04/2012

Recklessly Approximate Sparse Coding

It has recently been observed that certain extremely simple feature enco...
research
02/24/2022

Optimal Learning Rates of Deep Convolutional Neural Networks: Additive Ridge Functions

Convolutional neural networks have shown extraordinary abilities in many...
research
08/03/2021

Inference via Sparse Coding in a Hierarchical Vision Model

Sparse coding has been incorporated in models of the visual cortex for i...
research
04/27/2015

Linear Spatial Pyramid Matching Using Non-convex and non-negative Sparse Coding for Image Classification

Recently sparse coding have been highly successful in image classificati...
research
05/19/2017

Sparse Coding on Stereo Video for Object Detection

Deep Convolutional Neural Networks (DCNN) require millions of labeled tr...
research
11/21/2016

Training Sparse Neural Networks

Deep neural networks with lots of parameters are typically used for larg...
research
09/04/2017

Optimal deep neural networks for sparse recovery via Laplace techniques

This paper introduces Laplace techniques for designing a neural network,...

Please sign up or login with your details

Forgot password? Click here to reset