Skip to content
The methodOpen data
Back to the collection

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

2018 · arXiv

Language models
Read the paper
HN score
0
Mentions
20
Readers
20
Months
15
What’s in a score?

From the abstract

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

20 selected excerpts · 20 total mentions

thoughtpeddler
Neutral mention · rule
Thanks dang for compiling this. I suspect the Nov 2018 resurgence was due to Google publishing BERT around that time? The release of OpenAI’s GPT-1 [1] was earlier that year in June, so unlikely that. Of course Jan 2023 needs no explanation… And now in 2026 things are at a fever pitch. Interesting to trace these 10yr old AI posts from then to the present moment. The other one with a similar vintage would be “Should AI Be Open” [2] from Dec 2015, which is fascinating to juxtapose against the recent public battles. [0] “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding“: https://arxiv.org/abs/1810.04805 [1] “Improving Language Understanding by Generative Pre-Training”: https://cdn.openai.com/research-covers/language-unsupervised... [2] “Should AI Be Open?” | Slate Star Codex: https://slatestarcodex.com/2015/12/17/should-ai-be-open/
Original comment Superintelligence: The Idea That Eats Smart People (2016)
bhickey
Neutral mention · rule
The term 'GPT' was first used in the BERT paper to refer to Generic Pre-trained Transformers. [0] > The fine-tuning approach, such as the Generative Pre-trained Transformer (OpenAI GPT) (Radford et al., 2018), introduces minimal task-specific parameters, and is trained on the downstream tasks by simply fine-tuning all pre-trained parameters. [0] https://arxiv.org/abs/1810.04805
Original comment OpenAI – Application for US trademark “GPT” has failed
smoldesu
Neutral mention · rule
OpenAI was most certainly not the first organization to develop (or distribute) an LLM model: https://github.com/google-research/bert https://arxiv.org/abs/1810.04805 Failing that, Seq2seq predates OpenAI's incorporation: https://google.github.io/seq2seq/ https://github.com/google/seq2seq https://arxiv.org/abs/1409.3215
Original comment Ask HN: Why is OpenAI still called OpenAI?
PeterisP
Neutral mention · rule
The general idea is that you have a particular task & dataset, and you optimize these vectors to maximize that task. So the properties of these vectors - what information is retained and what is left out during the 'compression' - are effectively determined by that task. In general, the core task for the various "LLM tools" involves prediction of a hidden word, trained on very large quantities of real text - thus also mirroring whatever structure (linguistic, syntactic, semantic, factual, social bias, etc) exists there. If you want to see how the sausage is made and look at the actual algorithms, then the key two approaches to read up on would probably be Mikolov's…
Read the full comment
word2vec (https://arxiv.org/abs/1301.3781) with the CBOW (Continuous Bag of Words) and Continuous Skip-Gram Model, which are based on relatively simple math optimization, and then on the BERT (https://arxiv.org/abs/1810.04805) structure which does a conceptually similar thing but with a large neural network that can learn more from the same data. For both of them, you can either read the original papers or look up blog posts or videos that explain them, different people have different preferences on how readable academic papers are.
Original comment Embeddings: What they are and why they matter
NaiveBayesian
Neutral mention · rule
There was actually a series of language models named after Sesame Street characters back in 2018-2020, starting with ELMo, then BERT, ERNIE (a different model from 2019), Big Bird, ... There are likely some more that I missed. ELMo: https://arxiv.org/abs/1802.05365 BERT: https://arxiv.org/abs/1810.04805 ERNIE: https://arxiv.org/abs/1904.09223v1 Big Bird: https://arxiv.org/abs/2007.14062
Original comment Ernie, China's ChatGPT, Cracks Under Pressure
areyoukillingm3
Neutral mention · rule
Could you post a link to "the BERT paper"? I've read some, but would be interested reading anything that anyone considered definitive :) Is it this one? "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" :https://arxiv.org/abs/1810.04805
Original comment Ask HN: How to Break into AI Engineering
oersted
Neutral mention · rule
The achievement of training a BERT model to 90% of the GLUE score on a single GPU in ~100 hours is indeed impressive. As for the original BERT pretraining run, the paper [1] mentions that the pretraining took 4 days on 16 TPU chips for the BERT-Base model and 4 days on 64 TPU chips for the BERT-Large model. Regarding the translation of these techniques to the pretraining phase for a GPT model, it is possible that some of the optimizations and techniques used for BERT could be applied to GPT as well. However, the specific architecture and training objectives of GPT might require different approaches or additional optimizations. With the help of MirrorThink.ai, I accessed…
Read the full comment
the scientific papers to provide accurate information on the SOPHIA optimizer, which is designed to improve the training of deep learning models by adaptively adjusting the learning rate and momentum. According to the paper [2], SOPHIA has shown promising results in various deep learning tasks. It is possible that the SOPHIA optimizer could help improve the training of BERT and GPT models, but further research and experimentation would be needed to confirm its effectiveness in these specific cases. [1] https://arxiv.org/abs/1810.04805 [2] https://arxiv.org/pdf/2305.14342.pdf
Original comment Notes on training BERT from scratch on an 8GB consumer GPU
vikashrungta
Neutral mention · rule
I posted a list of papers on twitter, and will be posting a summary for each of them as well. here is the list https://twitter.com/vrungta/status/1623343807227105280 Unlocking the Secrets of AI: A Journey through the Foundational Papers by @vrungta (2023) 1. "Attention is All You Need" (2017) - https://arxiv.org/abs/1706.03762 (Google Brain) 2. "Generative Adversarial Networks" (2014) - https://arxiv.org/abs/1406.2661 (University of Montreal) 3. "Dynamic Routing Between Capsules" (2017) - https://arxiv.org/abs/1710.09829 (Google Brain) 4. "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks" (2016) -…
Read the full comment
https://arxiv.org/abs/1511.06434 (University of Montreal) 5. "ImageNet Classification with Deep Convolutional Neural Networks" (2012) - https://papers.nips.cc/paper/4824-imagenet-classification-wi... (University of Toronto) 6. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2018) - https://arxiv.org/abs/1810.04805 (Google) 7. "RoBERTa: A Robustly Optimized BERT Pretraining Approach" (2019) - https://arxiv.org/abs/1907.11692 (Facebook AI) 8. "ELMo: Deep contextualized word representations" (2018) - https://arxiv.org/abs/1802.05365 (Allen Institute for Artificial Intelligence) 9. "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context" (2019) - https://arxiv.org/abs/1901.02860 (Google AI Language) 10. "XLNet: Generalized Autoregressive Pretraining for Language Understanding" (2019) - https://arxiv.org/abs/1906.08237 (Google AI Language) 11. T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (2020) - https://arxiv.org/abs/1910.10683 (Google Research) 12. "Language Models are Few-Shot Learners" (2021) - https://arxiv.org/abs/2005.14165 (OpenAI)
Original comment What were the 40 research papers on the list Ilya Sutskever gave John Carmack?
lossolo
Neutral mention · rule
> Has Google/Alphabet publicly released any of their AI models yet? You mean NLP field changing models from Google like BERT [1]? or Transformers paper [2]? or T5 model [3] (used by company doing ChatGPT like search currently on the front page on HN)? 1. https://arxiv.org/abs/1810.04805 code+models: https://github.com/google-research/bert 2. https://arxiv.org/abs/2112.04426 3. https://arxiv.org/abs/1910.10683 code+models: https://github.com/google-research/text-to-text-transfer-tra...
Original comment AlphaCode Attention Visualization
svcrunch
Neutral mention · rule
This is an area that's dear to me, I'm the cofounder of Vectara and have been working with embedding-based semantic search, aka neural search or neural IR, since 2017. To whether Google uses semantic search, the answer is yes, very heavily [1][2]. Not only that, but they have led, and continue to lead, much of the pioneering research in NLP and neural IR for the past decade [3][4][5]. Technical challenges lie along a few primary dimensions. The first has been search quality, because, while early neural systems like Google Talk to Books [5][6] demonstrated the potential of these techniques, benchmarks like BEIR [7], released a few years later, in 2020, showed that the best…
Read the full comment
keyword retrieval algorithms still outperformed neural techniques in general settings. The landscape since then has shifted very rapidly: In 2022, for the first time, neural search methods outperformed BM25 on BEIR. This includes late interaction [8], sparse encoding [9], and, most challengingly, dense encoding [10] systems. The second technical challenge is scalability. After decades of infrastructure optimization, keyword systems scale well to very large corpora, while semantic systems struggle to achieve the same scale. The k-d tree approach presented in the article, for example, while good for experimentation, would be difficult to productionize, as-is, in a large-scale system. However, research into scaling dense vector retrieval has received a lot of focus recently [11], so I'm confident this will change. I'll close by saying your observation about being stuck with keyword search in a lot of apps is accurate, but I expect that to change soon. It's becoming easier to embed neural models everywhere, and I think that distilled models in the 5-50mb size range can feasibly power semantic search everywhere you press Ctrl-F today. [1] https://blog.google/products/search/search-language-understa... [2] https://blog.google/products/search/introducing-mum/ [2] https://arxiv.org/abs/1706.03762 [3] https://arxiv.org/abs/1810.04805 [4] https://arxiv.org/abs/1907.04307 [5] https://books.google.com/talktobooks/ [6] https://ai.googleblog.com/2018/04/introducing-semantic-exper... [7] https://arxiv.org/abs/2104.08663 [8] https://arxiv.org/abs/2112.01488 [9] https://arxiv.org/abs/2109.10086 [10] https://arxiv.org/pdf/2112.09118.pdf [11] https://www.microsoft.com/en-us/research/uploads/prod/2021/1...
Original comment Building a semantic search engine in Rust
tempodox
Neutral mention · rule
BERT engineer? That acronym is essentially ungooglable, since you just find every person ever named Bert. This is what I finally found: BERT, which stands for Bidirectional Encoder Representations from Transformers. https://arxiv.org/abs/1810.04805
Original comment Machine Learning: The Great Stagnation
ivalm
Neutral mention · rule
> xlnet trained on 10x more data than original BERT No, I've read on a github issue of xlnet that xlnet base is same size as bert base and xlnet large is same size as bert large. (I don't know for ernie 2) It's not about the size of the model, but the training data. If you read the XLNet paper https://arxiv.org/pdf/1906.08237.pdf they clearly state in section 3.1: "Following BERT [10], we use the BooksCorpus [41] and English Wikipedia as part of our pretraining data, which have 13GB plain text combined. In addition, we include Giga5 (16GB text) [23], ClueWeb 2012-B (extended from [5]), and Common Crawl [6] for pretraining. We use heuristics to aggressively filter out…
Read the full comment
short or low-quality articles for ClueWeb 2012-B and Common Crawl, which results in 19GB and 78GB text respectively. After tokenization with SentencePiece [16], we obtain 2.78B, 1.09B, 4.75B, 4.30B, and 19.97B subword pieces for Wikipedia, BooksCorpus, Giga5, ClueWeb, and Common Crawl respectively, which are 32.89B in total" If you compare to BERT paper https://arxiv.org/pdf/1810.04805.pdf training data for "pretraining data" section in also section 3.1: "Pre-training data The pre-training procedure largely follows the existing literature on language model pre-training. For the pre-training corpus we use the BooksCorpus (800M words) (Zhu et al., 2015) and English Wikipedia (2,500M words). For Wikipedia we extract only the text passages and ignore lists, tables, and headers. It is critical to use a document-level corpus rather than a shuffled sentence-level corpus such as the Billion Word Benchmark (Chelba et al., 2013) in order to extract long contiguous sequences." So 32.89B words for XLNet vs 3.3B words for BERT. We've also run fine tuning experiments supplementing additional private medical corpus (~10B words) and felt starting from clinical-bert was better than xlnet (for our rather specific use cases).
Original comment GPT-2: 6-Month Follow-Up
probably_wrong
Neutral mention · rule
> That's very ignorant to say that attention or convolution is anywhere close to the expressivity of RNNs(LSTM). But that's exactly the point of the Transformer model, with a paper aptly titled "Attention is all you need" [1]. And the Bert architecture, based in this idea, seems to be doing well. And they claim to be bery flexible, too[2]. Maybe that's what you meant with "unless you brutely search over hundreds of hyperparameters configs", but then again, isn't that what NNs are about anyway? [1] https://arxiv.org/abs/1706.03762 [2] https://arxiv.org/abs/1810.04805
Original comment Yann LeCun, Geoffrey Hinton and Yoshua Bengio win Turing Award
ColinWright
Neutral mention · rule
This desperately needs a "What is BERT" section. I've been reading for 5 minutes, skimming for 10, and I'm still not sure. Edit: OK, now I've clicked the correct random link and got the earlier paper: https://arxiv.org/pdf/1810.04805.pdf BERT is a "new language representation model". People, when you write papers, give your readers a chance to come up to speed - a list of links to previous reading is essential. Make it prominent, and early.
Original comment Dissecting Bert
solomatov
Neutral mention · rule
To summarize the achievements: * Attention is all you need transformer created a non recurrent architecture for NMT (https://arxiv.org/abs/1706.03762) * OpenAI GPT modified the original transformer by changing architectutre (one net instead of encoder/decoder pair), and using different hyperparameters which seems to work the best (https://s3-us-west-2.amazonaws.com/openai-assets/research-co...) * BERT used GPT's architecture but trained in a different way. Instead of training a language model, they forced the model predict holes in a text and predicting whether two sentences go one after another. (https://arxiv.org/abs/1810.04805) * OpenAI GPT2 achieved a new state of the art in language models (https://d4mucfpksywv.cloudfront.net/better-language-models/l...) * The paper in the top post found out that if we fine tune several models in the same way as in BERT, we get improvement in each of the fine tuned models.
Original comment Microsoft’s New MT-DNN Outperforms Google BERT
yoquan
Neutral mention · rule
Hmm, the article's summary table showed "only the top task layer(s)" for BERT fine-tuning. Is that correct? The original paper [1] wrote: "All of the parameters of BERT and W are fine-tuned jointly to maximize the log-probability of the correct label." [1]: https://arxiv.org/abs/1810.04805
Original comment Generalized Language Models

Keep following the thread.

More in Language models

More from the same topics.