research
∙
06/22/2022
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
We present the Pathways Autoregressive Text-to-Image (Parti) model, whic...
share
research
∙
12/13/2021
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Scaling language models with more data, compute and parameters has drive...
research
∙
10/09/2021
Vector-quantized Image Modeling with Improved VQGAN
Pretraining language models with next-token prediction on massive text c...
research
∙
09/27/2021
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
We summarize the results of a host of efforts using giant automatic spee...
research
∙
05/10/2021
GSPMD: General and Scalable Parallelization for ML Computation Graphs
We present GSPMD, an automatic, compiler-based parallelization system fo...
research
∙
11/07/2020
Exploring the limits of Concurrency in ML Training on Google TPUs
Recent results in language understanding using neural networks have requ...
research
∙
06/30/2020
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Neural network scaling has been critical for improving the model quality...
research
∙
04/28/2020
Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Training
In data-parallel synchronous training of deep neural networks, different...
research
∙
09/21/2019