Regularization Effect of Fast Gradient Sign Method and its Generalization
share
Fast Gradient Sign Method (FSGM) is a popular method to generate adversarial examples that make neural network models robust against perturbations. Despite its empirical success, its theoretical property is not well understood. This paper develops theory to explain the regularization effect of Generalized FSGM, a class of methods to generate adversarial examples. Motivated from the relationship between FSGM and LASSO penalty, the asymptotic properties of Generalized FSGM are derived in the Generalized Linear Model setting, which is essentially the 1-layer neural network setting with certain activation functions. In such simple neural network models, I prove that Generalized FSGM estimation is square root n-consistent and weakly oracle under proper conditions. The asymptotic results are also highly similar to penalized likelihood estimation. Nevertheless, Generalized FSGM introduces additional bias when data sampling is not sign neutral, a concept I introduce to describe the balanceness of the noise signs. Although the theory in this paper is developed under simple neural network settings, I argue that it may give insights and justification for FSGM in deep neural network settings as well.
READ FULL TEXT