Evolution of Activation Functions: An Empirical Investigation

by   Andrew Nader, et al.

The hyper-parameters of a neural network are traditionally designed through a time consuming process of trial and error that requires substantial expert knowledge. Neural Architecture Search (NAS) algorithms aim to take the human out of the loop by automatically finding a good set of hyper-parameters for the problem at hand. These algorithms have mostly focused on hyper-parameters such as the architectural configurations of the hidden layers and the connectivity of the hidden neurons, but there has been relatively little work on automating the search for completely new activation functions, which are one of the most crucial hyper-parameters to choose. There are some widely used activation functions nowadays which are simple and work well, but nonetheless, there has been some interest in finding better activation functions. The work in the literature has mostly focused on designing new activation functions by hand, or choosing from a set of predefined functions while this work presents an evolutionary algorithm to automate the search for completely new activation functions. We compare these new evolved activation functions to other existing and commonly used activation functions. The results are favorable and are obtained from averaging the performance of the activation functions found over 30 runs, with experiments being conducted on 10 different datasets and architectures to ensure the statistical robustness of the study.


page 1

page 2

page 3

page 4


Evolutionary Optimization of Deep Learning Activation Functions

The choice of activation function can have a large effect on the perform...

TanhSoft – a family of activation functions combining Tanh and Softplus

Deep learning at its core, contains functions that are composition of a ...

Data-Driven Learning of Feedforward Neural Networks with Different Activation Functions

This work contributes to the development of a new data-driven method (D-...

Normalized Activation Function: Toward Better Convergence

Activation functions are essential for neural networks to introduce non-...

Neural Networks with A La Carte Selection of Activation Functions

Activation functions (AFs), which are pivotal to the success (or failure...

An unfeasability view of neural network learning

We define the notion of a continuously differentiable perfect learning a...

The GatedTabTransformer. An enhanced deep learning architecture for tabular modeling

There is an increasing interest in the application of deep learning arch...

Please sign up or login with your details

Forgot password? Click here to reset