Parameterizing Activation Functions for Adversarial Robustness

by   Sihui Dai, et al.

Deep neural networks are known to be vulnerable to adversarially perturbed inputs. A commonly used defense is adversarial training, whose performance is influenced by model capacity. While previous works have studied the impact of varying model width and depth on robustness, the impact of increasing capacity by using learnable parametric activation functions (PAFs) has not been studied. We study how using learnable PAFs can improve robustness in conjunction with adversarial training. We first ask the question: how should we incorporate parameters into activation functions to improve robustness? To address this, we analyze the direct impact of activation shape on robustness through PAFs and observe that activation shapes with positive outputs on negative inputs and with high finite curvature can increase robustness. We combine these properties to create a new PAF, which we call Parametric Shifted Sigmoidal Linear Unit (PSSiLU). We then combine PAFs (including PReLU, PSoftplus and PSSiLU) with adversarial training and analyze robust performance. We find that PAFs optimize towards activation shape properties found to directly affect robustness. Additionally, we find that while introducing only 1-2 learnable parameters into the network, smooth PAFs can significantly increase robustness over ReLU. For instance, when trained on CIFAR-10 with additional synthetic data, PSSiLU improves robust accuracy by 4.54 WRN-28-10 in the ℓ_∞ threat model while adding only 2 additional parameters into the network architecture. The PSSiLU WRN-28-10 model achieves 61.96 on RobustBench (Croce et al., 2020).


page 1

page 2

page 3

page 4


SPLASH: Learnable Activation Functions for Improving Accuracy and Adversarial Robustness

We introduce SPLASH units, a class of learnable activation functions sho...

Smooth Adversarial Training

It is commonly believed that networks cannot be both accurate and robust...

Low Curvature Activations Reduce Overfitting in Adversarial Training

Adversarial training is one of the most effective defenses against adver...

Parametric Rectified Power Sigmoid Units: Learning Nonlinear Neural Transfer Analytical Forms

The paper proposes representation functionals in a dual paradigm where l...

Robust Principles: Architectural Design Principles for Adversarially Robust CNNs

Our research aims to unify existing works' diverging opinions on how arc...

Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality

Adversarial training is a popular method to give neural nets robustness ...

Limitations of Piecewise Linearity for Efficient Robustness Certification

Certified defenses against small-norm adversarial examples have received...

Please sign up or login with your details

Forgot password? Click here to reset