An Evolutionary Algorithm of Linear complexity: Application to Training of Deep Neural Networks

07/12/2019
by   S. Ivvan Valdez, et al.
18

The performance of deep neural networks, such as Deep Belief Networks formed by Restricted Boltzmann Machines (RBMs), strongly depends on their training, which is the process of adjusting their parameters. This process can be posed as an optimization problem over n dimensions. However, typical networks contain tens of thousands of parameters, making this a High-Dimensional Problem (HDP). Although different optimization methods have been employed for this goal, the use of most of the Evolutionary Algorithms (EAs) becomes prohibitive due to their inability to deal with HDPs. For instance, the Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) which is regarded as one of the most effective EAs, exhibits the enormous disadvantage of requiring O(n^2) memory and operations, making it unpractical for problems with more than a few hundred variables. In this paper, we introduce a novel EA that requires O(n) operations and memory, but delivers competitive solutions for the training stage of RBMs with over one million variables, when compared against CMA-ES and the Contrastive Divergence algorithm, which is the standard method for training RBMs.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/09/2015

Training Restricted Boltzmann Machines via the Thouless-Anderson-Palmer Free Energy

Restricted Boltzmann machines are undirected neural networks which have ...
research
05/15/2019

Regularized Evolutionary Algorithm for Dynamic Neural Topology Search

Designing neural networks for object recognition requires considerable a...
research
05/19/2018

GADAM: Genetic-Evolutionary ADAM for Deep Neural Network Optimization

Deep neural network learning can be formulated as a non-convex optimizat...
research
04/26/2018

Optimal-margin evolutionary classifier

We introduce a novel approach for discriminative classification using ev...
research
03/28/2018

Supervising Unsupervised Learning with Evolutionary Algorithm in Deep Neural Network

A method to control results of gradient descent unsupervised learning in...
research
06/26/2018

Limited Evaluation Evolutionary Optimization of Large Neural Networks

Stochastic gradient descent is the most prevalent algorithm to train neu...
research
03/18/2020

Block Layer Decomposition schemes for training Deep Neural Networks

Deep Feedforward Neural Networks' (DFNNs) weights estimation relies on t...

Please sign up or login with your details

Forgot password? Click here to reset