Scaling Limit: Exact and Tractable Analysis of Online Learning Algorithms with Applications to Regularized Regression and PCA

by   Chuang Wang, et al.

We present a framework for analyzing the exact dynamics of a class of online learning algorithms in the high-dimensional scaling limit. Our results are applied to two concrete examples: online regularized linear regression and principal component analysis. As the ambient dimension tends to infinity, and with proper time scaling, we show that the time-varying joint empirical measures of the target feature vector and its estimates provided by the algorithms will converge weakly to a deterministic measured-valued process that can be characterized as the unique solution of a nonlinear PDE. Numerical solutions of this PDE can be efficiently obtained. These solutions lead to precise predictions of the performance of the algorithms, as many practical performance metrics are linear functionals of the joint empirical measures. In addition to characterizing the dynamic performance of online learning algorithms, our asymptotic analysis also provides useful insights. In particular, in the high-dimensional limit, and due to exchangeability, the original coupled dynamics associated with the algorithms will be asymptotically "decoupled", with each coordinate independently solving a 1-D effective minimization problem via stochastic gradient descent. Exploiting this insight for nonconvex optimization problems may prove an interesting line of future research.


page 1

page 2

page 3

page 4


The Scaling Limit of High-Dimensional Online Independent Component Analysis

We analyze the dynamics of an online algorithm for independent component...

Subspace Estimation from Incomplete Observations: A High-Dimensional Analysis

We present a high-dimensional analysis of three popular algorithms, name...

A General Framework for Analyzing Stochastic Dynamics in Learning Algorithms

We present a general framework for analyzing high-probability bounds for...

Online Learning Algorithms for Quaternion ARMA Model

In this paper, we address the problem of adaptive learning for autoregre...

Asymptotic linear expansion of regularized M-estimators

Parametric high-dimensional regression analysis requires the usage of re...

Asymptotically Efficient Online Learning for Censored Regression Models Under Non-I.I.D Data

The asymptotically efficient online learning problem is investigated for...

FADO: A Deterministic Detection/Learning Algorithm

This paper proposes and studies a detection technique for adversarial sc...

Please sign up or login with your details

Forgot password? Click here to reset