research
∙
05/13/2023
Depth Dependence of μP Learning Rates in ReLU MLPs
In this short note we consider random fully connected ReLU networks of w...
share
research
∙
12/01/2022
Differentially Private Adaptive Optimization with Delayed Preconditioners
Privacy noise may negate the benefits of using adaptive optimizers in di...
research
∙
10/12/2022
Large Models are Parsimonious Learners: Activation Sparsity in Trained Transformers
This paper studies the curious phenomenon for machine learning models wi...
research
∙
02/12/2022
Private Adaptive Optimization with Side Information
Adaptive optimization methods have become the default solvers for many m...
research
∙
02/02/2022
Robust Training of Neural Networks using Scale Invariant Architectures
In contrast to SGD, adaptive gradient methods like Adam allow robust tra...
research
∙
02/13/2021
Distilling Double Descent
Distillation is the technique of training a "student" model based on exa...
research
∙
08/08/2020
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
Federated learning is a challenging optimization problem due to the hete...
research
∙
06/08/2020
O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformers
Transformer networks use pairwise attention to compute contextual embedd...
research
∙
05/21/2020
Why distillation helps: a statistical perspective
Knowledge distillation is a technique for improving the performance of a...
research
∙
04/23/2020
Doubly-stochastic mining for heterogeneous retrieval
Modern retrieval problems are characterised by training sets with potent...
research
∙
02/17/2020
Low-Rank Bottleneck in Multi-head Attention Models
Attention based Transformer architecture has enabled significant advance...
research
∙
12/20/2019
Are Transformers universal approximators of sequence-to-sequence functions?
Despite the widespread adoption of Transformer models for NLP tasks, the...
research
∙
12/06/2019
Why ADAM Beats SGD for Attention Models
While stochastic gradient descent (SGD) is still the de facto algorithm ...
research
∙
10/14/2019
SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning
Federated learning is a key scenario in modern large-scale machine learn...
research
∙
08/20/2019
AdaCliP: Adaptive Clipping for Private SGD
Privacy preserving machine learning algorithms are crucial for learning ...
research
∙
04/19/2019
On the Convergence of Adam and Beyond
Several recently proposed stochastic optimization methods that have been...
research
∙
01/26/2019
Escaping Saddle Points with Adaptive Gradient Methods
Adaptive methods such as Adam and RMSProp are widely used in deep learni...
research
∙
10/16/2018
Stochastic Negative Mining for Learning with Large Output Spaces
We consider the problem of retrieving the most relevant labels for a giv...
research
∙
09/05/2017
A Generic Approach for Escaping Saddle points
A central challenge to using first-order methods for optimizing nonconve...
research
∙
08/24/2016
AIDE: Fast and Communication Efficient Distributed Optimization
In this paper, we present two new communication-efficient methods for di...
research
∙
07/27/2016
Stochastic Frank-Wolfe Methods for Nonconvex Optimization
We study Frank-Wolfe methods for nonconvex stochastic and finite-sum opt...
research
∙
05/23/2016
Fast Stochastic Methods for Nonsmooth Nonconvex Optimization
We analyze stochastic algorithms for optimizing nonconvex, nonsmooth fin...
research
∙
03/19/2016
Stochastic Variance Reduction for Nonconvex Optimization
We study nonconvex finite-sum problems and analyze stochastic variance r...
research
∙
03/19/2016
Fast Incremental Method for Nonconvex Optimization
We analyze a fast incremental aggregated gradient method for optimizing ...
research
∙
08/04/2015
Adaptivity and Computation-Statistics Tradeoffs for Kernel and Distance based High Dimensional Two Sample Testing
Nonparametric two sample testing is a decision theoretic problem that in...
research
∙
06/23/2015
On Variance Reduction in Stochastic Gradient Descent and its Asynchronous Variants
We study optimization algorithms based on variance reduction for stochas...
research
∙
11/23/2014
On the High-dimensional Power of Linear-time Kernel Two-Sample Testing under Mean-difference Alternatives
Nonparametric two sample testing deals with the question of consistently...
research
∙
10/16/2012