Improving Robust Generalization by Direct PAC-Bayesian Bound Minimization

by   Zifan Wang, et al.

Recent research in robust optimization has shown an overfitting-like phenomenon in which models trained against adversarial attacks exhibit higher robustness on the training set compared to the test set. Although previous work provided theoretical explanations for this phenomenon using a robust PAC-Bayesian bound over the adversarial test error, related algorithmic derivations are at best only loosely connected to this bound, which implies that there is still a gap between their empirical success and our understanding of adversarial robustness theory. To close this gap, in this paper we consider a different form of the robust PAC-Bayesian bound and directly minimize it with respect to the model posterior. The derivation of the optimal solution connects PAC-Bayesian learning to the geometry of the robust loss surface through a Trace of Hessian (TrH) regularizer that measures the surface flatness. In practice, we restrict the TrH regularizer to the top layer only, which results in an analytical solution to the bound whose computational cost does not depend on the network depth. Finally, we evaluate our TrH regularization approach over CIFAR-10/100 and ImageNet using Vision Transformers (ViT) and compare against baseline adversarial robustness algorithms. Experimental results show that TrH regularization leads to improved ViT robustness that either matches or surpasses previous state-of-the-art approaches while at the same time requires less memory and computational cost.


page 1

page 2

page 3

page 4


A PAC-Bayes Analysis of Adversarial Robustness

We propose the first general PAC-Bayesian generalization bounds for adve...

Adversarial Robustness through Local Linearization

Adversarial training is an effective methodology for training deep neura...

PAC-Bayesian Transportation Bound

We present a new generalization error bound, the PAC-Bayesian transporta...

Second Order Optimization for Adversarial Robustness and Interpretability

Deep neural networks are easily fooled by small perturbations known as a...

Improving Robustness using Generated Data

Recent work argues that robust training requires substantially larger da...

Open Category Detection with PAC Guarantees

Open category detection is the problem of detecting "alien" test instanc...

On the Adversarial Robustness of Mixture of Experts

Adversarial robustness is a key desirable property of neural networks. I...

Please sign up or login with your details

Forgot password? Click here to reset