Enhance Diffusion to Improve Robust Generalization

by   Jianhui Sun, et al.

Deep neural networks are susceptible to human imperceptible adversarial perturbations. One of the strongest defense mechanisms is Adversarial Training (AT). In this paper, we aim to address two predominant problems in AT. First, there is still little consensus on how to set hyperparameters with a performance guarantee for AT research, and customized settings impede a fair comparison between different model designs in AT research. Second, the robustly trained neural networks struggle to generalize well and suffer from tremendous overfitting. This paper focuses on the primary AT framework - Projected Gradient Descent Adversarial Training (PGD-AT). We approximate the dynamic of PGD-AT by a continuous-time Stochastic Differential Equation (SDE), and show that the diffusion term of this SDE determines the robust generalization. An immediate implication of this theoretical finding is that robust generalization is positively correlated with the ratio between learning rate and batch size. We further propose a novel approach, Diffusion Enhanced Adversarial Training (DEAT), to manipulate the diffusion term to improve robust generalization with virtually no extra computational burden. We theoretically show that DEAT obtains a tighter generalization bound than PGD-AT. Our empirical investigation is extensive and firmly attests that DEAT universally outperforms PGD-AT by a significant margin.


page 1

page 2

page 3

page 4


An SDE Framework for Adversarial Training, with Convergence and Robustness Analysis

Adversarial training has gained great popularity as one of the most effe...

Certified Robust Neural Networks: Generalization and Corruption Resistance

Adversarial training aims to reduce the problematic susceptibility of mo...

Stability Analysis and Generalization Bounds of Adversarial Training

In adversarial machine learning, deep neural networks can fit the advers...

Max-Margin Adversarial (MMA) Training: Direct Input Space Margin Maximization through Adversarial Training

We propose Max-Margin Adversarial (MMA) training for directly maximizing...

Enhancing Adversarial Training with Second-Order Statistics of Weights

Adversarial training has been shown to be one of the most effective appr...

You Only Propagate Once: Accelerating Adversarial Training Using Maximal Principle

Deep learning achieves state-of-the-art results in many areas. However r...

The Devil's Advocate: Shattering the Illusion of Unexploitable Data using Diffusion Models

Protecting personal data against the exploitation of machine learning mo...

Please sign up or login with your details

Forgot password? Click here to reset