A Constructive Approach for One-Shot Training of Neural Networks Using Hypercube-Based Topological Coverings

by   W. Brent Daniel, et al.

In this paper we presented a novel constructive approach for training deep neural networks using geometric approaches. We show that a topological covering can be used to define a class of distributed linear matrix inequalities, which in turn directly specify the shape and depth of a neural network architecture. The key insight is a fundamental relationship between linear matrix inequalities and their ability to bound the shape of data, and the rectified linear unit (ReLU) activation function employed in modern neural networks. We show that unit cover geometry and cover porosity are two design variables in cover-constructive learning that play a critical role in defining the complexity of the model and generalizability of the resulting neural network classifier. In the context of cover-constructive learning, these findings underscore the age old trade-off between model complexity and overfitting (as quantified by the number of elements in the data cover) and generalizability on test data. Finally, we benchmark on algorithm on the Iris, MNIST, and Wine dataset and show that the constructive algorithm is able to train a deep neural network classifier in one shot, achieving equal or superior levels of training and test classification accuracy with reduced training time.


Improving Classification Neural Networks by using Absolute activation function (MNIST/LeNET-5 example)

The paper discusses the use of the Absolute activation function in class...

A New Constructive Method to Optimize Neural Network Architecture and Generalization

In this paper, after analyzing the reasons of poor generalization and ov...

Concentration inequalities and optimal number of layers for stochastic deep neural networks

We state concentration and martingale inequalities for the output of the...

Understanding Deep Neural Networks Using Topological Data Analysis

Deep neural networks (DNN) are black box algorithms. They are trained us...

Multiclass classification for multidimensional functional data through deep neural networks

The intrinsically infinite-dimensional features of the functional observ...

Improving learnability of neural networks: adding supplementary axes to disentangle data representation

Over-parameterized deep neural networks have proven to be able to learn ...

Continuously Constructive Deep Neural Networks

Traditionally, deep learning algorithms update the network weights where...

Please sign up or login with your details

Forgot password? Click here to reset