The Sample Complexity of Dictionary Learning

by   Daniel Vainsencher, et al.

A large set of signals can sometimes be described sparsely using a dictionary, that is, every element can be represented as a linear combination of few elements from the dictionary. Algorithms for various signal processing applications, including classification, denoising and signal separation, learn a dictionary from a set of signals to be represented. Can we expect that the representation found by such a dictionary for a previously unseen example from the same source will have L_2 error of the same magnitude as those for the given examples? We assume signals are generated from a fixed distribution, and study this questions from a statistical learning theory perspective. We develop generalization bounds on the quality of the learned dictionary for two types of constraints on the coefficient selection, as measured by the expected L_2 error in representation when the dictionary is used. For the case of l_1 regularized coefficient selection we provide a generalization bound of the order of O(sqrt(np log(m lambda)/m)), where n is the dimension, p is the number of elements in the dictionary, lambda is a bound on the l_1 norm of the coefficient vector and m is the number of samples, which complements existing results. For the case of representing a new signal as a combination of at most k dictionary elements, we provide a bound of the order O(sqrt(np log(m k)/m)) under an assumption on the level of orthogonality of the dictionary (low Babel function). We further show that this assumption holds for most dictionaries in high dimensions in a strong probabilistic sense. Our results further yield fast rates of order 1/m as opposed to 1/sqrt(m) using localized Rademacher complexity. We provide similar results in a general setting using kernels with weak smoothness requirements.


page 1

page 2

page 3

page 4


Minimax Lower Bounds for Kronecker-Structured Dictionary Learning

Dictionary learning is the problem of estimating the collection of atomi...

On The Sample Complexity of Sparse Dictionary Learning

In the synthesis model signals are represented as a sparse combinations ...

Finding GEMS: Multi-Scale Dictionaries for High-Dimensional Graph Signals

Modern data introduces new challenges to classic signal processing appro...

On the Sample Complexity of Predictive Sparse Coding

The goal of predictive sparse coding is to learn a representation of exa...

A Structured Dictionary Perspective on Implicit Neural Representations

Propelled by new designs that permit to circumvent the spectral bias, im...

Separation of undersampled composite signals using the Dantzig selector with overcomplete dictionaries

In many applications one may acquire a composition of several signals th...

Local identifiability of l_1-minimization dictionary learning: a sufficient and almost necessary condition

We study the theoretical properties of learning a dictionary from N sign...

Please sign up or login with your details

Forgot password? Click here to reset