Attention is All you Need

That paper is using a pre-existing term "Generative Pretraining" [0] and applying it to Transformers (a Google innovation [1]). As far as I can see from a search, they don't even use the term GPT or Generative Pretrained Transformers in that paper, and they don't in the accompanying blog post either [2]. A sibling [3] claims that the BERT paper in Oct 2018 was the first to use the term GPT to describe what OpenAI built, and that sounds reasonable since a cursory look through the Sep 2018 archive of openai.com turns up nothing. [0] See this 2012 example: http://cs224d.stanford.edu/papers/maas_paper.pdf [1] https://proceedings.neurips.cc/paper/2017/file/3f5ee243547de... [2] http://web.archive.org/web/20180923011305/https://blog.opena... [3] https://news.ycombinator.com/item?id=39381802

I think they mean Transformers in the Vaswani et al 'Attention is all you need' paper, not Generative Pretrained Transformers, specifically? Paper link below: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547de... For some papers on attention mechanisms from before the 2017 'Attention is all you need' paper, check out that paper's references. Chris Manning's 2015 paper covers attention mechanisms. And so do a few other researchers from that mid-2010s time period: [21] Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention based neural machine translation. arXiv preprint arXiv:1508.04025, 2015. [22] Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model. In Empirical Methods in Natural Language Processing, 2016.

For a while now, an answer I've seen is to start with "Attention Is All You Need", the original Transformers paper. It's still pretty good, but over the past year I've led a few working sessions on grokking transformer computational fundamentals and they've turned up some helpful later additions that simplify and clarify what's going on. You can quickly get overwhelmed by the million good resources out there so I'll keep it to these three. If you have a strong CS background, they'll take you a long way: (1) Transformers from Scratch: https://peterbloem.nl/blog/transformers (2) Attention Is All You Need: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547de... (3) Formal Algorithms for Transformers: https://arxiv.org/abs/2207.09238

For those who are interested in how Transformers seem more prevalent, please read this thread by Karpathy where he talks about a consolidation in ML: https://twitter.com/karpathy/status/1468370605229547522 And of course one of the early classic papers in the field, as a bonus: https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd05... (The paper is mentioned in the article)

No, they took a pre-existing term "Generative Pretraining"[0] and applied it to Transformers (a Google innovation [1]) to get Generative Pretrained Transformers [2]. Even if you looked at the full name, that paper doesn't use GPT or Generative Pretrained Transformers at all from what I can tell, this commenter [3] claims that the name was first used in the BERT paper. [0] See this 2012 example: http://cs224d.stanford.edu/papers/maas_paper.pdf [1] https://proceedings.neurips.cc/paper/2017/file/3f5ee243547de... [2] https://cdn.openai.com/research-covers/language-unsupervised... [3] https://news.ycombinator.com/item?id=39381802

I agree that the copycats are wearing thin, but the original paper's title seems fine to me. It's an accurate description of the breakthrough they made. The first few sentences of the Attention is All You Need abstract explain it pretty well: > The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547de...