Enhancing Adversarial Training with Second-Order Statistics of Weights

by   Gaojie Jin, et al.
University of Liverpool

Adversarial training has been shown to be one of the most effective approaches to improve the robustness of deep neural networks. It is formalized as a min-max optimization over model weights and adversarial perturbations, where the weights can be optimized through gradient descent methods like SGD. In this paper, we show that treating model weights as random variables allows for enhancing adversarial training through Second-Order Statistics Optimization (S^2O) with respect to the weights. By relaxing a common (but unrealistic) assumption of previous PAC-Bayesian frameworks that all weights are statistically independent, we derive an improved PAC-Bayesian adversarial generalization bound, which suggests that optimizing second-order statistics of weights can effectively tighten the bound. In addition to this theoretical insight, we conduct an extensive set of experiments, which show that S^2O not only improves the robustness and generalization of the trained neural networks when used in isolation, but also integrates easily in state-of-the-art adversarial training techniques like TRADES, AWP, MART, and AVMixup, leading to a measurable improvement of these techniques. The code is available at <https://github.com/Alexkael/S2O>.


page 6

page 14


Adversarial Robustness through Regularization: A Second-Order Approach

Adversarial training is a common approach to improving the robustness of...

Randomized Adversarial Training via Taylor Expansion

In recent years, there has been an explosion of research into developing...

Distributed Adversarial Training to Robustify Deep Neural Networks at Scale

Current deep neural networks (DNNs) are vulnerable to adversarial attack...

On the distance between two neural networks and the stability of learning

How far apart are two neural networks? This is a foundational question i...

Robustness, Privacy, and Generalization of Adversarial Training

Adversarial training can considerably robustify deep neural networks to ...

A Robust Optimization Approach to Deep Learning

Many state-of-the-art adversarial training methods leverage upper bounds...

Enhance Diffusion to Improve Robust Generalization

Deep neural networks are susceptible to human imperceptible adversarial ...

Please sign up or login with your details

Forgot password? Click here to reset