Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT
share
We combine the capacity of sparsely gated Mixture-of-Experts (MoE) with the speed and stability of linear, mixing transformations to design the Sparse Mixer encoder model. The Sparse Mixer slightly outperforms (<1 and SuperGLUE, but more importantly trains 65 faster. We also present a faster variant, prosaically named Fast Sparse Mixer, that marginally underperforms (<0.2 nearly twice as fast: 89 the design of these two models by carefully ablating through various mixing mechanisms, MoE configurations and model hyperparameters. The Sparse Mixer overcomes many of the latency and stability concerns of MoE models and offers the prospect of serving sparse student models, without resorting to distilling them to dense variants.
READ FULL TEXT