Formal Algorithms for Transformers

If you want the actual maths instead of handwaving, I recommend: https://arxiv.org/abs/2207.09238 For something gentler, 3Blue1Brown: https://www.youtube.com/watch?v=eMlx5fFNoYc (this is part of a series)

Not a core paper, but I found Formal Algorithms for Transformers [1] (a Google paper from 2022) to have a great pedagogical style. [1] https://arxiv.org/abs/2207.09238

Tangential, but: the notes that this post refers to are probably not the best way to learn how transformers work. If you want mathematical precision, those notes are based on this paper from DeepMind: https://arxiv.org/abs/2207.09238 The paper provides mathematically precise definitions of all the parts of a transformers, though it's showing its age (ha!) in that it doesn't include some formalizations that are common in, for example, Llama.

> the paper that was most helpful for me was [Formal Algorithms for Transformers](https://arxiv.org/abs/2207.09238) Interesting but hard to read since it uses a quite unique notations for matrix indexing and multplication. Why???

'Formal Algorithms for Transformers'[1] is a proper account of the architectures and what tasks they naturally lend themselves to, by authors from DeepMind. See sections 3 (Transformers and Typical Tasks) and 6 (Transformer Architectures). Not much on empirical observations, though. [1]https://arxiv.org/abs/2207.09238

https://arxiv.org/abs/2207.09238 This is a great transparent and clear description of what is actually going on in attention layers. Alternatively, this article by Jay Alammar is also very popular http://jalammar.github.io/illustrated-transformer/

I've been working through [0]. Like a lot of math, the notation is daunting, but once you become familiar with it, it really is a nice tool for thought. [0]: https://arxiv.org/abs/2207.09238

Here are a few resources that might help get you started: 1. A thread explaining the internal working of transformers: https://twitter.com/hippopedoid/status/1641432291149848576?s... 2. Paper by DeepMind which provides pseudo-code for important algorithms for Transformer models: https://arxiv.org/pdf/2207.09238.pdf 3. Another thread specifically on large language models: https://twitter.com/cwolferesearch/status/164044611134855577... Once again these are not courses per se, but do provide intuitive explanations for how transformers work. There is also the nanoGPT series of videos by Karpathy on youtube. First video here: https://www.youtube.com/watch?v=kCc8FmEb1nY

For a while now, an answer I've seen is to start with "Attention Is All You Need", the original Transformers paper. It's still pretty good, but over the past year I've led a few working sessions on grokking transformer computational fundamentals and they've turned up some helpful later additions that simplify and clarify what's going on. You can quickly get overwhelmed by the million good resources out there so I'll keep it to these three. If you have a strong CS background, they'll take you a long way: (1) Transformers from Scratch: https://peterbloem.nl/blog/transformers (2) Attention Is All You Need: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547de... (3) Formal Algorithms for Transformers: https://arxiv.org/abs/2207.09238

An early explainer of transformers, which is a quicker read, that I found very useful when they were still new to me, is The Illustrated Transformer[1], by Jay Alammar. A more recent academic but high-level explanation of transformers, very good for detail on the different flow flavors (e.g. encoder-decoder vs decoder only), is Formal Algorithms for Transformers[2], from DeepMind. [1] https://jalammar.github.io/illustrated-transformer/ [2] https://arxiv.org/abs/2207.09238

I've been reading this paper with pseudocode for various transformers and finding it helfpul: https://arxiv.org/abs/2207.09238 "This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (not results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models."

I've been reading this paper with pseudocode for various transformers and finding it helfpul: https://arxiv.org/abs/2207.09238 "This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (not results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models."