research
∙
07/13/2023
Accelerated gradient methods for nonconvex optimization: Escape trajectories from strict saddle points and convergence to local minima
This paper considers the problem of understanding the behavior of a gene...
share
research
∙
05/20/2023
Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descent
Algorithmic stability is an important notion that has proven powerful fo...
research
∙
02/10/2023
Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD
Cyclic and randomized stepsizes are widely used in the deep learning pra...
research
∙
01/27/2023
Algorithmic Stability of Heavy-Tailed SGD with General Loss Functions
Heavy-tail phenomena in stochastic gradient descent (SGD) have been repo...
research
∙
01/16/2023
Distributionally Robust Learning with Weakly Convex Losses: Convergence Rates and Finite-Sample Guarantees
We consider a distributionally robust stochastic optimization problem an...
research
∙
11/29/2022
Penalized Langevin and Hamiltonian Monte Carlo Algorithms for Constrained Sampling
We consider the constrained sampling problem where the goal is to sample...
research
∙
06/02/2022
Algorithmic Stability of Heavy-Tailed Stochastic Gradient Descent on Least Squares
Recent studies have shown that heavy tails can emerge in stochastic opti...
research
∙
05/13/2022
Heavy-Tail Phenomenon in Decentralized SGD
Recent theoretical studies have shown that heavy-tails can emerge in sto...
research
∙
08/20/2021
L-DQN: An Asynchronous Limited-Memory Distributed Quasi-Newton Method
This work proposes a distributed algorithm for solving empirical risk mi...
research
∙
06/09/2021
Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms
Understanding generalization in deep learning has been one of the major ...
research
∙
06/07/2021
TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversion
This work proposes a time-efficient Natural Gradient Descent method, cal...
research
∙
02/20/2021
Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variance
Recent studies have provided both empirical and theoretical evidence ill...
research
∙
02/13/2021
Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
Gaussian noise injections (GNIs) are a family of simple and widely-used ...
research
∙
08/05/2020
Differentially Private Accelerated Optimization Algorithms
We present two classes of differentially private optimization algorithms...
research
∙
07/01/2020
Decentralized Stochastic Gradient Langevin Dynamics and Hamiltonian Monte Carlo
Stochastic gradient Langevin dynamics (SGLD) and stochastic gradient Ham...
research
∙
06/11/2020
IDEAL: Inexact DEcentralized Accelerated Augmented Lagrangian Method
We introduce a framework for designing primal methods under the decentra...
research
∙
06/08/2020
A Stochastic Subgradient Method for Distributionally Robust Non-Convex Learning
We consider a distributionally robust formulation of stochastic optimiza...
research
∙
06/08/2020
The Heavy-Tail Phenomenon in SGD
In recent years, various notions of capacity and complexity have been pr...
research
∙
05/25/2020
Fractional moment-preserving initialization schemes for training fully-connected neural networks
A common approach to initialization in deep neural networks is to sample...
research
∙
04/06/2020
Non-Convex Stochastic Optimization via Non-Reversible Stochastic Gradient Langevin Dynamics
Stochastic gradient Langevin dynamics (SGLD) is a poweful algorithm for ...
research
∙
02/13/2020
Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noise
Stochastic gradient descent with momentum (SGDm) is one of the most popu...
research
∙
11/29/2019
On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
The gradient noise (GN) in the stochastic gradient descent (SGD) algorit...
research
∙
06/21/2019
First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise
Stochastic gradient descent (SGD) has been widely used in machine learni...
research
∙
01/23/2019
A Universally Optimal Multistage Accelerated Stochastic Gradient Method
We study the problem of minimizing a strongly convex and smooth function...
research
∙
01/22/2019
Accelerated Linear Convergence of Stochastic Momentum Methods in Wasserstein Distances
Momentum methods such as Polyak's heavy ball (HB) method, Nesterov's acc...
research
∙
01/18/2019
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
The gradient noise (GN) in the stochastic gradient descent (SGD) algorit...
research
∙
12/19/2018
Breaking Reversibility Accelerates Langevin Dynamics for Global Non-Convex Optimization
Langevin dynamics (LD) has been proven to be a powerful technique for op...
research
∙
09/12/2018
Global Convergence of Stochastic Gradient Hamiltonian Monte Carlo for Non-Convex Stochastic Optimization: Non-Asymptotic Performance Bounds and Momentum-Based Acceleration
Stochastic gradient Hamiltonian Monte Carlo (SGHMC) is a variant of stoc...
research
∙
10/24/2017