research
∙
05/10/2021
GSPMD: General and Scalable Parallelization for ML Computation Graphs
We present GSPMD, an automatic, compiler-based parallelization system fo...
share
research
∙
11/07/2020
Exploring the limits of Concurrency in ML Training on Google TPUs
Recent results in language understanding using neural networks have requ...
research
∙
06/30/2020
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Neural network scaling has been critical for improving the model quality...
research
∙
04/28/2020
Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Training
In data-parallel synchronous training of deep neural networks, different...
research
∙
09/21/2019
Scale MLPerf-0.6 models on Google TPU-v3 Pods
The recent submission of Google TPU-v3 Pods to the industry wide MLPerf ...
research
∙
02/21/2019
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Lingvo is a Tensorflow framework offering a complete solution for collab...
research
∙
11/16/2018
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
GPipe is a scalable pipeline parallelism library that enables learning o...
research
∙
11/05/2018