Skip to content
The methodOpen data
Back to the collection

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Zhenzhong Lan, Mingda Chen, Sebastian Goodman

Show all 6 authors

Kevin Gimpel, Piyush Sharma, Radu Soricut

2019 · arXiv

Language modelsMachine learning
Read the paper
HN score
0
Mentions
4
Readers
4
Months
3
What’s in a score?

From the abstract

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at https://github.com/google-research/ALBERT.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

albertzeyer
Neutral mention · rule
Nice, thanks for naming a project after me. :P I also like the ALBERT lite-BERT model (https://arxiv.org/abs/1909.11942). Although this was just a matter of time. The other names which match to "*bert*" were already taken. Anyway, going on topic: Such a project really should have a related work section, right at the beginning, because there are many existing launchers, and just say how this is different (maybe better). E.g. see here (https://alternativeto.net/software/alfred/?platform=linux). But I also would be interested on differences e.g. to Alfred (Mac). I really wonder why such comparisons are very often missing. Also, a bit more information would be nice. Is this…
Read the full comment
cross platform? Or Linux only? It seems Mac is on the todo-list (https://github.com/albertlauncher/albert/issues/1050) but I don't see a Mac binary download, so I assume it's not ready yet? Although the issue was closed. Btw, the last commit is from on 9 Oct 2021. So maybe already abandoned?
Original comment Albert – open-source keyboard launcher for Linux
danieldk
Neutral mention · rule
There's nothing preventing you from sharing weights across layers, and would be interesting to see some research about that. E.g. the ALBERT model does that: https://arxiv.org/abs/1909.11942 I have done model distillation of XLM-RoBERTa into ALBERT-based models with multiple layer groups and for the tasks that I was working on (syntax) it works really well. E.g. we have gone from a finetuned ~1000MiB XLM-R base model to a 74MiB ALBERT-based model with barely any loss in accuracy.
Original comment Scaling Down Deep Learning
aglionby
Neutral mention · rule
Paper here: https://arxiv.org/abs/1909.11942 Really impressed at the speed with which Hugging Face ported this to their transformers library -- Google released the model and source code Oct 21 [1] and it was available in the library just 8 days later [2]. [1] https://github.com/google-research/google-research/commit/b5... [2] https://github.com/huggingface/transformers/commit/c0c208833...
Original comment ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations

Keep following the thread.

More in Language models

More from the same topics.