Nearly Optimal Algorithms with Sublinear Computational Complexity for Online Kernel Regression

by   Junfan Li, et al.

The trade-off between regret and computational cost is a fundamental problem for online kernel regression, and previous algorithms worked on the trade-off can not keep optimal regret bounds at a sublinear computational complexity. In this paper, we propose two new algorithms, AOGD-ALD and NONS-ALD, which can keep nearly optimal regret bounds at a sublinear computational complexity, and give sufficient conditions under which our algorithms work. Both algorithms dynamically maintain a group of nearly orthogonal basis used to approximate the kernel mapping, and keep nearly optimal regret bounds by controlling the approximate error. The number of basis depends on the approximate error and the decay rate of eigenvalues of the kernel matrix. If the eigenvalues decay exponentially, then AOGD-ALD and NONS-ALD separately achieves a regret of O(√(L(f))) and O(d_eff(μ)lnT) at a computational complexity in O(ln^2T). If the eigenvalues decay polynomially with degree p≥ 1, then our algorithms keep the same regret bounds at a computational complexity in o(T) in the case of p>4 and p≥ 10, respectively. L(f) is the cumulative losses of f and d_eff(μ) is the effective dimension of the problem. The two regret bounds are nearly optimal and are not comparable.


page 1

page 2

page 3

page 4


Improved Kernel Alignment Regret Bound for Online Kernel Learning

In this paper, we improve the kernel alignment regret bound for online k...

Scalable Thompson Sampling using Sparse Gaussian Process Models

Thompson Sampling (TS) with Gaussian Process (GP) models is a powerful t...

Mixability made efficient: Fast online multiclass logistic regression

Mixability has been shown to be a powerful tool to obtain algorithms wit...

Second-Order Kernel Online Convex Optimization with Adaptive Sketching

Kernel online convex optimization (KOCO) is a framework combining the ex...

Nearly Optimal Risk Bounds for Kernel K-Means

In this paper, we study the statistical properties of the kernel k-means...

Context-aware Dynamic Assets Selection for Online Portfolio Selection based on Contextual Bandit

Online portfolio selection is a sequential decision-making problem in fi...

A nearly optimal explicitly-sparse representation for oscillatory kernels with curvelet-like functions

A nearly optimal explicitly-sparse representation for oscillatory kernel...

Please sign up or login with your details

Forgot password? Click here to reset