Skip to content
The methodOpen data
Back to the collection

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

William Fedus, Barret Zoph, Noam Shazeer

2021 · arXiv

Language modelsAlgorithmsNetworks
Read the paper
HN score
0
Mentions
6
Readers
6
Months
6
What’s in a score?

From the abstract

In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs and training instability -- we address these with the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques help wrangle the instabilities and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the "Colossal Clean Crawled Corpus" and achieve a 4x speedup over the T5-XXL model.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

6 selected excerpts · 6 total mentions

radq
Neutral mention · rule
If it's similar to the switch transformer architecture [1], which I suspect it is, then the models are all trained on the same corpus and the routing model learns automatically which experts to route to. It's orthogonal to beam search - the benefit of the architecture is that it allows sparse inference. [1] https://arxiv.org/pdf/2101.03961.pdf
Original comment GPT4 is 8 x 220B params = 1.7T params
espadrine
Neutral mention · rule
My educated guess is that they use a MoE-style model similar to the Switch transformer[0], and combine a similar encoding as that of Kosmos-1[1] (with an “image” latch token, and a ViT-style transformer to process images). As a result, the parameter count is likely bigger, but since not all of them are involved in a forward pass, it is not as meaningful. [0]: https://arxiv.org/pdf/2302.14045.pdf [1]: https://arxiv.org/pdf/2101.03961.pdf
Original comment GPT-4
jayalammar
Neutral mention · rule
The size of the database, the training set, the details of the architecture, as well as results on benchmark tasks should all be considered in the comparison. I'm also a fan of Behavioral Testing [1]. Parameter count is not very accurate measure of model performance. Mixture of Expert models like the Switch Transformer [2] can be 1 trillion parameters in size, but are not 5X the performance, for example. They clock the retrieval at 10 ms, unclear if that includes the BERT inference, however. My assumption is that it does not. [1] https://arxiv.org/abs/2005.04118 [2] https://arxiv.org/abs/2101.03961
Original comment The Illustrated Retrieval Transformer
criticaltinker
Neutral mention · rule
First, congratulations to you and the other authors - this is a very informative and accessible paper with impressive results. > On both datasets, we find that synthesis performance scales log-linearly with model size This "scaling law" observation is starting to become a major trend in NLP [1] and other other modalities such as speech recognition [2] and protein structure/function prediction [3]. Do you have any insight or commentary to offer regarding the next step in improving program synthesis? For example, will techniques to scale up model size continue to be a primary focus (eg as in [4]), or do you see improvements in Transformer and attention based architectures…
Read the full comment
as essential for pushing the limits of what has been achieved by you and your colleagues? > We find that even our best models are generally unable to predict the output of a program given a specific input. What do you think about leveraging unsupervised training to improve program synthesis? Could synthesized programs be executed on generated input in a way that supports contrastive learning [5]? Thanks in advance for your time and comments here. [1] Scaling Laws for Neural Language Models https://arxiv.org/abs/2001.08361 [2] Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition https://arxiv.org/abs/2010.10504 [3] ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning https://arxiv.org/abs/2007.06225 [4] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity https://arxiv.org/abs/2101.03961 [5] A Simple Framework for Contrastive Learning of Visual Representations https://arxiv.org/pdf/2108.00587.pdf
Original comment Program Synthesis with Large Language Models

Keep following the thread.

More in Language models

More from the same topics.