An Experimental Study of Weight Initialization and Weight Inheritance Effects on Neuroevolution

by   Zimeng Lyu, et al.

Weight initialization is critical in being able to successfully train artificial neural networks (ANNs), and even more so for recurrent neural networks (RNNs) which can easily suffer from vanishing and exploding gradients. In neuroevolution, where evolutionary algorithms are applied to neural architecture search, weights typically need to be initialized at three different times: when initial genomes (ANN architectures) are created at the beginning of the search, when offspring genomes are generated by crossover, and when new nodes or edges are created during mutation. This work explores the difference between using Xavier, Kaiming, and uniform random weight initialization methods, as well as novel Lamarckian weight inheritance methods for initializing new weights during crossover and mutation operations. These are examined using the Evolutionary eXploration of Augmenting Memory Models (EXAMM) neuroevolution algorithm, which is capable of evolving RNNs with a variety of modern memory cells (e.g., LSTM, GRU, MGU, UGRNN and Delta-RNN cells) as well recurrent connections with varying time skips through a high performance island based distributed evolutionary algorithm. Results show that with statistical significance, utilizing the Lamarckian strategies outperforms Kaiming, Xavier and uniform random weight initialization, and can speed neuroevolution by requiring less backpropagation epochs to be evaluated for each generated RNN.


page 1

page 2

page 3

page 4


An Empirical Exploration of Deep Recurrent Connections and Memory Cells Using Neuro-Evolution

Neuro-evolution and neural architecture search algorithms have gained in...

Accelerating the Evolution of Convolutional Neural Networks with Node-Level Mutations and Epigenetic Weight Initialization

This paper examines three generic strategies for improving the performan...

Investigating Recurrent Neural Network Memory Structures using Neuro-Evolution

This paper presents a new algorithm, Evolutionary eXploration of Augment...

Evolving A-Type Artificial Neural Networks

We investigate Turing's notion of an A-type artificial neural network. W...

Improving Neuroevolution Using Island Extinction and Repopulation

Neuroevolution commonly uses speciation strategies to better explore the...

The Ant Swarm Neuro-Evolution Procedure for Optimizing Recurrent Networks

Hand-crafting effective and efficient structures for recurrent neural ne...

Safe Mutations for Deep and Recurrent Neural Networks through Output Gradients

While neuroevolution (evolving neural networks) has a successful track r...

Please sign up or login with your details

Forgot password? Click here to reset