Taming the heavy-tailed features by shrinkage and clipping
share
In this paper, we consider the generalized linear models (GLM) with heavy-tailed features and corruptions. Besides clipping the response, we propose to shrink the feature vector by its ℓ_4-norm under the low dimensional regime and clip each entry of the feature vector in the high-dimensional regime. Under bounded fourth moment assumptions, we show that the MLE based on shrunk or clipped data enjoys nearly the minimax optimal rate with exponential deviation bound. Simulations demonstrate significant improvement in statistical performance by feature shrinkage and clipping in linear regression with heavy-tailed noise and logistic regression with noisy labels. We also apply shrinkage to deep features of MNIST images and find that classifiers trained by shrunk deep features are fairly robust to noisy labels: it achieves 0.9% testing error in the presence of 40% mislabeled data.
READ FULL TEXT