E-swish: Adjusting Activations to Different Network Depths

01/22/2018

∙

by   Eric Alcaide, et al.

∙

1

∙

share

Activation functions have a notorious impact on neural networks on both training and testing the models against the desired problem. Currently, the most used activation function is the Rectified Linear Unit (ReLU). This paper introduces a new and novel activation function, closely related with the new activation Swish = x * sigmoid(x) (Ramachandran et al., 2017) which generalizes it. We call the new activation E-swish = β x * sigmoid(x). We show that E-swish outperforms many other well-known activations including both ReLU and Swish. For example, using E-swish provided 1.5 improvements on Cifar10 and Cifar100 respectively for the WRN 10-2 when compared to ReLU and 0.35 code to reproduce all our experiments can be found at https://github.com/EricAlcaide/E-swish