Skip to content
The methodOpen data
Back to the collection

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar

Show all 8 authors

Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

2017 · arXiv

Language modelsMachine learningComputer vision
Read the paper
HN score
1.5
Mentions
83
Readers
76
Months
46
What’s in a score?

From the abstract

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

20 selected excerpts · 83 total mentions

seanhunter
Strong recommendation · rule
I genuinely don't know what you mean by "blind probability" or "surface-level probability", but in case you don't know, how transformer models work is after tokenization and embeddings are calculated each attention head in the model calculates "self-attention" which is the relationship between all tokens in the current context of the model and passes its results either to another layer of attention heads or to the decoder layer of the model. This typically uses a temperature parameter to control randomness in the output by affecting the probability distribution of tokens in the vocabulary when producing output. I'm not sure whether this counts as "blind" or "surface-level" but that's the probability in GPT models. If you want to learn more, here's Vaswani et al 2017 - the seminal paper https://arxiv.org/pdf/1706.03762.pdf
Original comment ChatGPT: It has no inner monologue or meta-cognition
nsagent
Neutral mention · rule
I'm sorry, but I'm not sure what works best for a beginner. I started my PhD when the original Transformers paper [1] had just been released. I had no background in NLP whatsoever and used the original paper to write a Transformer and the full training pipeline from scratch during the first couple months of my PhD without referencing any existing code (only reading the paper and it's references). So I'd say, if you're motivated you could do the same. That said, I've always been a self-starter and I started my PhD after working for a decade. I'm sure there are other resources out there, but I'm not equipped to say what's best for a beginner (I found the original paper to…
Read the full comment
be excellent, but most everyone during my PhD, including my advisor, found it to be inscrutable; I think it's written more like an engineering focused paper, which might be why researchers found it difficult to grok, but with my previous industry experience it seemed quite clear). [1]: https://arxiv.org/abs/1706.03762
Original comment Apple Sues OpenAI for Stealing Trade Secrets to Build AI Hardware
cute_boi
Neutral mention · rule
No need stupid moving texts. CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/ The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Understanding LSTM Networks - https://colah.github.io/posts/2015-08-Understanding-LSTMs/ ImageNet Classification with Deep Convolutional Neural Networks - https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... Deep Residual Learning for Image Recognition - https://arxiv.org/abs/1512.03385 Multi-Scale Context Aggregation by Dilated Convolutions - https://arxiv.org/abs/1511.07122 Identity Mappings in Deep Residual Networks -…
Read the full comment
https://arxiv.org/abs/1603.05027 Recurrent Neural Network Regularization - https://arxiv.org/abs/1409.2329 Deep Speech 2: End-to-End Speech Recognition in English and Mandarin - https://arxiv.org/abs/1512.02595 Order Matters: Sequence to Sequence for Sets - https://arxiv.org/abs/1511.06391 Neural Machine Translation by Jointly Learning to Align and Translate - https://arxiv.org/abs/1409.0473 Pointer Networks - https://arxiv.org/abs/1506.03134 Attention Is All You Need - https://arxiv.org/abs/1706.03762 The Annotated Transformer - https://nlp.seas.harvard.edu/annotated-transformer/ Neural Turing Machines - https://arxiv.org/abs/1410.5401 A Simple Neural Network Module for Relational Reasoning - https://arxiv.org/abs/1706.01427 Relational Recurrent Neural Networks - https://arxiv.org/abs/1806.01822 Neural Message Passing for Quantum Chemistry - https://arxiv.org/abs/1704.01212 Scaling Laws for Neural Language Models - https://arxiv.org/abs/2001.08361 GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism - https://arxiv.org/abs/1811.06965 Keeping Neural Networks Simple by Minimizing the Description Length of the Weights - https://www.cs.toronto.edu/~hinton/absps/colt93.pdf A Tutorial Introduction to the Minimum Description Length Principle - https://arxiv.org/abs/math/0406077 The First Law of Complexodynamics - https://scottaaronson.blog/?p=762 Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton - https://arxiv.org/abs/1405.6903 Kolmogorov Complexity - https://onlinelibrary.wiley.com/doi/book/10.1002/047174882X Variational Lossy Autoencoder - https://arxiv.org/abs/1611.02731 Machine Super Intelligence - https://www.vetta.org/documents/Machine_Super_Intelligence.p...
Original comment 30papers.com – Ilya's 30 essential ML papers, in a beginner friendly format
benoau
Neutral mention · rule
There's actually eight authors of that paper, if ChatGPT is accurate: - Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones and Lukasz Kaiser stayed on at Google - Aidan N. Gomez became a researcher - Illia Polosukhin co-founded https://near.ai https://arxiv.org/abs/1706.03762
Original comment Ask HN: What happened to the guy that invented LLMs?
lyfeninja
Neutral mention · rule
Below is the "Attention is all you need" paper. Transformers and their attention mechanism was the major breakthrough for modern LLMs. ML has been around for a long time, I'd suggest joining kaggle or something and learn by doing. You'll retain more and realize how broad the category is anymore. https://arxiv.org/abs/1706.03762
Original comment Ask HN: A Brief History of LLMs
newelh
Neutral mention · rule
I built udoc because most document extraction tools I've used require significant dependencies, only handle one format, or have restrictive licenses. I wanted a single binary that reads PDFs, Office docs (including legacy .doc/.xls/.ppt), ODF, and RTF — with no external parsers, no system packages, nothing to install. It's written in pure Rust with Python bindings via PyO3. If you have uv, you can try it right now without installing anything: `curl -sL https://arxiv.org/pdf/1706.03762 \ | uvx udoc - | grep -A 18 '^Abstract'` Highlights: A CLI: e.g. udoc -J ingest.pdf | duckdb -c "COPY (SELECT * FROM read_json_auto('/dev/stdin')) TO 'pages.parquet'". One unified Document…
Read the full comment
model across all formats: extracted documents are organized into 5 layers - Content, Metadata, Presentation, Relationships, Interactions. Streaming page-by-page extraction, so a 10 GB PDF doesn't need to fit in memory. A JSONL-based hook protocol for plugging in OCR (Tesseract, cloud APIs), layout detection (DocLayout-YOLO), or vision-language models as subprocesses. PDF rendering engine "udoc render paper.pdf -o ./pages" Typed diagnostics enable recoverable issues like font fallbacks or malformed xref tables are structured warnings you can filter on. A frequent question: if udoc is a full document toolkit, why does it not include OCR? Because OCR is not a parser; it is a model that reconstructs text from pixels. No parser can substitute for it. The relevant question is whether the parser knows when to invoke it. udoc's approach: Automatic scan detection. Pages with one large image, fewer than five text spans, and no extractable glyph data are flagged as LikelyScanned on the diagnostics sink. The OCR hook fires only on those pages by default. OCR as a hook, not a built-in. Tesseract, GLM-OCR, DeepSeek-OCR, Textract, Document AI, Azure Form Recognizer: the right engine depends on the document, the language, the hardware, the budget, and the data-egress policy. udoc does not ship one. The hook protocol lets you wire whichever engine you need. Per-page granularity. The detector runs per page, not per document. OCR fires on scanned inserts and skips the digitally-generated body. This is an alpha release. APIs and output format may still change. The docs are at https://newelh.github.io/udoc/ if you want to go deeper. Happy to answer questions about the parsing approach, format quirks, or anything else.
Original comment Show HN: udoc. Dependency-free document extraction in Rust
make3
Neutral mention · rule
The transformer paper was 9 years ago. 9 years between barely translating alright between two very closely related languages (English and French, huge fraction of shared words because of William the conqueror and cultural proximity etc etc) and what we have now. The thing is able to code up full pretty competent thousand lines projects in an hour. Even hardcore engineers use it now, as of this year. My senior front end friends already can't find jobs. You're crazy if you think things won't change dramatically, at the scale of all of society. https://arxiv.org/abs/1706.03762
Original comment Claude for Creative Work
ninjagoo
Neutral mention · rule
> Superior architectures will leak pretty quickly via engineers. I agree with the outcome of your premise (i.e., openness), but for different reasons: First, isn't it the case that these bleeding edge 'newfangled' LLMs are basically variations on the same core ideas from "Attention Is All You Need" from 2017? [1]. Different scale, but still the same basic architecture. Even the "MoE" innovation keeps the Transformer attention stack while replacing or augmenting the dense feed-forward/MLP part with routed expert blocks. And, I would argue that Engineers aren't working on new architectures. That would be Researchers, working on State-space models/Mamba (CMU/Princeton…
Read the full comment
ecosystem), Diffusion Language Models (Inception Labs), Long-convolution architectures/Hyena (Stanford etc.), RWKV/Recurrent LLMs (open-source community), Memory-augmented architectures (Google Research/DeepMind?), World models/spatial intelligence (LeCun/Fei-Fei Li/DeepMind), Symbolic/neurosymbolic alternatives, Thousand brains (Numenta). That research is still open, so the outcome that you propose (openness) is likely to come to pass. Researchers/Scientists gotta publish, otherwise it's not science (to quote LeCun [2]) [1] https://arxiv.org/abs/1706.03762 [2] https://x.com/ylecun/status/1795589846771147018
Original comment Open Weights Kill the Moat
smokel
Neutral mention · rule
Interestingly, the person at Microsoft states in a reply that even most of them have to pick this up in their spare time. Judging from other replies, it seems that there are quite some differences in how companies approach this. What I meant with "menial coding" is those jobs where people have to submit TPS reports on how much hours they spend for each customer. Reading a paper such as this one [1] is typically not directly necessary for being a good frontend developer, but it might stimulate someone to develop into a more fruitful employee in the long run. Managers would have to explain to their customer why time is being spent on that, and that requires some vision and creativity, which is not always a given. [1] https://arxiv.org/abs/1706.03762
Original comment Five Years of Running a Systems Reading Group at Microsoft
max_
Neutral mention · rule
No particular order but; 1. Transformers I think they need to be looked into more. They just work too well. I feel like they have a secret within that is yet to be known. [1] https://arxiv.org/abs/1706.03762 2. The 2016 Noble Prize in Medicine Autophagy has such interesting properties. It can probably do alot more for us now than any new pill or gene editing tool [2] https://www.nobelprize.org/prizes/medicine/2016/press-releas... 3. zk-STARKS I can prove to you than X * Y = Z, without ever revealing what, X, Y or Z are. And I can do this for any function, or math operator. https://eprint.iacr.org/2023/189 4. TLock I can encrypt my will, in a way such that. It can only be read after X years. And I do not need a trusted third party to do it. [4] https://eprint.iacr.org/2018/046.pdf [4] https://youtu.be/Xh849Ij3lhU
Original comment Ask HN: What are the greatest discoveries in the last few years?
Kim_Bruning
Neutral mention · rule
LLMs were -in part- designed as translation tools. It's one thing they do really really well. https://arxiv.org/html/1706.03762v7 (Attention is all you need) "Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train." Ok, looking that up, that was quite literally one of the main design goals. And they're really quite good at translating between the languages I use. They're the best tool for the job.
Original comment Don't post generated/AI-edited comments. HN is for conversation between humans.
canyon289
Neutral mention · rule
Hi, I'm not on the t5 Gemma team but work on gemma in general. Encoder Decoder comes from the original transformers implementation way back in 2017. If you look at figure 1 you'll see what the first transformer ever looked like. Since that time different implementations of transformers use either just the encoder portion, or the decoder portion, or both. Its a deep topic so hard to summarize here, but Gemini explains it really well! Hope this gets you started on some prompting to learn more https://arxiv.org/pdf/1706.03762
Original comment T5Gemma 2: The next generation of encoder-decoder models
dormento
Neutral mention · rule
In case anyone missed the reference: https://arxiv.org/abs/1706.03762 > (...) We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
Original comment LLMs can get "brain rot"
hodgesrm
Neutral mention · rule
> You're missing the point. In the past bad writing was just bad writing, and it was typically easy to detect. If AI generated text were well written, would it matter to you? Is it bad to use Grammarly? I don't see anything inherently wrong with using AI tools to write, as long as writers take the responsibility to ensure the final result is good. Fighting against use of LLMs seems like a fool's errand at this point. Personally I've been using Google Translate for years to help with writing in German, little knowing at the time that it was using transformers under the covers. [0] I'm pretty sure my correspondents would have thanked me had they known. Same applies for text written in English by non-native speakers. [0] https://arxiv.org/abs/1706.03762 edit: fixed typo. Just proof this is not an LLM.
Original comment The great software quality collapse or, how we normalized catastrophe
tankenmate
Neutral mention · rule
The title of this paper is a reference to a previous paper titled "Attention Is All You Need"[0][1]. This seminal work described the transformer model that is the basis for almost all LLMs, and is almost certainly the most cited paper on AI even though it was only published in 2017. [0] https://arxiv.org/abs/1706.03762 [1] https://en.wikipedia.org/wiki/Attention_Is_All_You_Need
Original comment TransMLA: Multi-head latent attention is all you need
albertzeyer
Neutral mention · rule
It's really a pity that they do this now. Some of their older papers had actually quite some valuable information, comments, discussions, thoughts, even commented out sections, figures, tables in it. It gave a much better view on how the paper was written over time, or how even the work processed over time. Sometimes you also see some alternative titles being discussed, which can be quite funny. E.g. from https://arxiv.org/abs/1804.09849: %\title{Sequence-to-Sequence Tricks and Hybrids\\for Improved Neural Machine Translation} % \title{Mixing and Matching Sequence-to-Sequence Modeling Techniques\\for Improved Neural Machine Translation} % \title{Analyzing and Optimizing…
Read the full comment
Sequence-to-Sequence Modeling Techniques\\for Improved Neural Machine Translation} % \title{Frankenmodels for Improved Neural Machine Translation} % \title{Optimized Architectures and Training Strategies\\for Improved Neural Machine Translation} % \title{Hybrid Vigor: Combining Traits from Different Architectures Improves Neural Machine Translation} \title{The Best of Both Worlds: \\Combining Recent Advances in Neural Machine Translation\\ ~} Also a lot of things in the Attention is all you need paper: https://arxiv.org/abs/1706.03762v1
Original comment ArXiv LaTeX Cleaner: Clean the LaTeX code of your paper to submit to ArXiv
leopoldj
Neutral mention · rule
This is a faithful reproduction of the original Transformer paper [1]. Except, these days we use trainable parameters for positional embedding. The paper used a static calculation for positional embedding using sine and cosine. Figure 1 in the paper can be seen implemented in the forward() method of the GPT class in model.py. Here are the rough steps: 1. Tokens are embedded using a nn.Embedding layer. 2. Tokens are positionally embedded using a nn.Embedding layer. 3. The two embedding values are added to make the input x. 4. A sequence of N number of transformer blocks are then executed. This is the grey box in the left of the Figure 1. This is where all the magic…
Read the full comment
happens. Chiefly in the self attention calculation. You can see this in the forward() method of the CausalSelfAttention class. 5. A regular nn.Linear layer is executed. 6. Finally the output token probabilities are calculated using F.cross_entropy (shown as softmax in the figure). I hope this helps a little. Please feel free to suggest improvements and additions. [1] https://arxiv.org/pdf/1706.03762
Original comment SmolGPT: A minimal PyTorch implementation for training a small LLM from scratch

Keep following the thread.

More in Language models

More from the same topics.