On the Sample Complexity of Predictive Sparse Coding

by   Nishant A. Mehta, et al.

The goal of predictive sparse coding is to learn a representation of examples as sparse linear combinations of elements from a dictionary, such that a learned hypothesis linear in the new representation performs well on a predictive task. Predictive sparse coding algorithms recently have demonstrated impressive performance on a variety of supervised tasks, but their generalization properties have not been studied. We establish the first generalization error bounds for predictive sparse coding, covering two settings: 1) the overcomplete setting, where the number of features k exceeds the original dimensionality d; and 2) the high or infinite-dimensional setting, where only dimension-free bounds are useful. Both learning bounds intimately depend on stability properties of the learned sparse encoder, as measured on the training sample. Consequently, we first present a fundamental stability result for the LASSO, a result characterizing the stability of the sparse codes with respect to perturbations to the dictionary. In the overcomplete setting, we present an estimation error bound that decays as Õ(sqrt(d k/m)) with respect to d and k. In the high or infinite-dimensional setting, we show a dimension-free bound that is Õ(sqrt(k^2 s / m)) with respect to k and s, where s is an upper bound on the number of non-zeros in the sparse code for any training data point.


page 1

page 2

page 3

page 4


Dimensionality-Dependent Generalization Bounds for k-Dimensional Coding Schemes

The k-dimensional coding schemes refer to a collection of methods that a...

Sparse coding for multitask and transfer learning

We investigate the use of sparse coding and dictionary learning in the c...

The Sample Complexity of Dictionary Learning

A large set of signals can sometimes be described sparsely using a dicti...

Dictionary Learning with BLOTLESS Update

Algorithms for learning a dictionary under which a data in a given set h...

Provably Accurate Double-Sparse Coding

Sparse coding is a crucial subroutine in algorithms for various signal p...

Adversarial Robustness of Supervised Sparse Coding

Several recent results provide theoretical insights into the phenomena o...

Sensitivity of sparse codes to image distortions

Sparse coding has been proposed as a theory of visual cortex and as an u...

Please sign up or login with your details

Forgot password? Click here to reset