Learn to Enhance the Negative Information in Convolutional Neural Network

by   Zhicheng Cai, et al.

This paper proposes a learnable nonlinear activation mechanism specifically for convolutional neural network (CNN) termed as LENI, which learns to enhance the negative information in CNNs. In sharp contrast to ReLU which cuts off the negative neurons and suffers from the issue of ”dying ReLU”, LENI enjoys the capacity to reconstruct the dead neurons and reduce the information loss. Compared to improved ReLUs, LENI introduces a learnable approach to process the negative phase information more properly. In this way, LENI can enhance the model representational capacity significantly while maintaining the original advantages of ReLU. As a generic activation mechanism, LENI possesses the property of portability and can be easily utilized in any CNN models through simply replacing the activation layers with LENI block. Extensive experiments validate that LENI can improve the performance of various baseline models on various benchmark datasets by a clear margin (up to 1.24 on ImageNet-1k) with negligible extra parameters. Further experiments show that LENI can act as a channel compensation mechanism, offering competitive or even better performance but with fewer learned parameters than baseline models. In addition, LENI introduces the asymmetry to the model structure which contributes to the enhancement of representational capacity. Through visualization experiments, we validate that LENI can retain more information and learn more representations.


Reborn Mechanism: Rethinking the Negative Phase Information Flow in Convolutional Neural Network

This paper proposes a novel nonlinear activation mechanism typically for...

Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units

Recently, convolutional neural networks (CNNs) have been used as a power...

EraseReLU: A Simple Way to Ease the Training of Deep Convolution Neural Networks

For most state-of-the-art architectures, Rectified Linear Unit (ReLU) be...

Activate or Not: Learning Customized Activation

Modern activation layers use non-linear functions to activate the neuron...

An Enhanced Convolutional Neural Network in Side-Channel Attacks and Its Visualization

In recent years, the convolutional neural networks (CNNs) have received ...

What do CNN neurons learn: Visualization Clustering

In recent years convolutional neural networks (CNN) have shown striking ...

First Power Linear Unit with Sign

This paper proposes a novel and insightful activation method termed FPLU...

Please sign up or login with your details

Forgot password? Click here to reset