Skip to content
The methodOpen data
Back to the collection

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov

Show all 12 authors

Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby

2020 · arXiv

Computer visionLanguage modelsPsychology
Read the paper
HN score
0
Mentions
14
Readers
13
Months
11
What’s in a score?

From the abstract

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

14 selected excerpts · 14 total mentions

ninjha
Neutral mention · rule
I think I remember one of the original ViT papers saying something about 2D embeddings on image patches not actually increasing performance on image recognition or segmentation, so it’s kind of interesting that it helps with text! E: I found the paper: https://arxiv.org/pdf/2010.11929 > We use standard learnable 1D position embeddings, since we have not observed significant performance gains from using more advanced 2D-aware position embeddings (Appendix D.4). Although it looks like that was just ImageNet so maybe this isn't that surprising.
Original comment Baba Is Eval
GistNoesis
Neutral mention · rule
(2020) https://arxiv.org/abs/2010.11929 : an image is worth 16x16 words transformers for image recognition at scale (2021) https://arxiv.org/abs/2103.13915 : An Image is Worth 16x16 Words, What is a Video Worth? (2024) https://arxiv.org/abs/2406.07550 : An Image is Worth 32 Tokens for Reconstruction and Generation
Original comment Sora is here
llm_nerd
Neutral mention · rule
>This is not at all how this works. You can literally convert the tokens returned from a video to text. What do you even think tokens are? Like seriously, before you write another word on this feel free to call the API and retrieve tokens for a video or image. Now go through the magical process of converting those tokens back to their text form. It isn't some magical hyper-dimensional, inside-out spatial encoding that yields impossible compression. This process is obvious and logical if actually thought through. >Each image is about 258 tokens Because Google set that as the "budget" and truncates accordingly. Again, call the API with an image or video and then convert those tokens to text. >https://arxiv.org/abs/2010.11929 This is super weird, and does not remotely prove your point. I literally spend most of my days in ViTs, but thanks for the link.
Original comment The killer app of Gemini Pro 1.5 is using video as an input
famouswaffles
Neutral mention · rule
This is not at all how this works. There's no separate model. Yes there's unique tokenization, if not the video as a whole then for each image. The whole video is ~1800 tokens because Gemini gets video as a series of images in context at 1 frame/s. Each image is about 258 tokens because a token in image transformer terms is literally a patch of the image. https://arxiv.org/abs/2010.11929
Original comment The killer app of Gemini Pro 1.5 is using video as an input
godelski
Neutral mention · rule
> The field of ML research is moving so fast that people don’t even take time anymore to explain the design decisions behind their architectures. Has it ever? I've read a lot of "old" papers and idk if there's ever been a strong theoretical framework. I'd say it is strongest today but not in main papers and those papers are getting rejected because of lack of experiments (wtf is going on with reviewing and why are our solutions "authors have to show reviewers used LLMs"[-1] not "reviewers have to provide meaningful review"? What a joke!). Here's a more recent Universal Approximation paper[0]. Cyberko paper[1] is good, but I think we're moving in the right direction, at…
Read the full comment
least with respect to these two works. Other UA papers I've seen are pretty handwavy. But simple also doesn't mean "not useful" or "not worthy of publish" and I think this is something people are forgetting lately. Probably because we're asked to review too much and annoyed, but don't take it out on authors. Some examples are GELU[2] (read the end, there's drama in this paper), Clean-FID (CVPR 22)[3], ViT (ICLR 21) and DeiT (ICML 21)[4,5], and even ResNet (CVPR 16)[6] and tons of ReLU papers we can list. But let's also not confuse big compute with "good work" (they correlate but aren't the same. You can definitely do more with more compute). > he problem with this for me is that we fail to build a nice, crisp understanding of the effects of each design decision in the final outcomes, which hurts the actual “science” of it. It also opens up the field for bogus and unreproducible claims. This is mostly an unsolvable problem in science and I think we actually have bigger problems. This can be solved without conferences due to our community being fairly good about open sourcing code and checkpoints (at least compared to any other research area), as well as just the papers themselves. Reproducibility is only able to be done when people can actually replicate works (lol sorry GPT papers...). What helps this is moving away from publish or perish or allowing people to publish works that are replication attempts (replication is the foundation of science. Why does everything have to be "novel", whatever that means). The bigger problem I see is actually related to the above. Publishing is just insane. It is easy, especially in fast moving times, to claim any work is incremental or not novel. That it's "just x but y". My feelings are "so what?" ViT is "just a transformer on images but image patches are tokens," yet it's insanely useful. It's obvious post hoc but not a priori. If it was, it would already be in use (ViT actually being a good example given the timeline). Everything is simpler and much more obvious after you've already seen it, which is why the whole notion of novelty is ridiculous. We humans just rewrite history in our minds and we also only concentrate on what's popular (lots of incentives, especially with reviewing insanity). Poor Ross Wightman gets shadowed for replications and improving ResNets though (and poor ConvNext). But we're seeing big labs do this and get through because they can do lots of experiments and build a lot of empirical evidence towards work, despite no additional theory. The problem? What about the academics? I don't know how people publish without connections to big labs (aka big compute) anymore. The ideas from the groups are the same (industry is a bit more hyped), but I've seen papers where the theory is good, the architecture makes sense (and is well explained) and my co-reviewers for a fucking workshop want to reject it because "not enough experiments" and "only one 'real world' experiment." These are papers that are on par or better than many I see in the main conferences (you can tell they're shifted to workshops due to rejects). Talk about holding back science. Holding back science is making authors submit works over and over and over to get orthogonal reviews every time. Holding back science is me arguing with my co-reviewers who admitted to not understanding the work that "more experiments" is not a sufficient reason to reject from a workshop (or a main conference). Hell, I got rejected once for a distillation paper and the reviewers complained that my teacher model's performance didn't match that of the original paper (no one has replicated that work btw, and I beat the second best published result I could find). Another paper I got hard rejected because I redacted a github link and an appendix citation broke when splitting the work. Guess what the ACs did? They let that person write a stronger response and then because they were the only high confidence reviewer on that paper (others were 3/5 confidence and borderline/WA) and so rejected the work. What kind of insane system is that? I see instances like this all over the place from many different peers across many different universities. There's more that's broken before we can even begin to talk about other issues because our system of "what qualifies as meaningful work" shouldn't be a fucking slot machine run by undergrads. CVPR has well over 10k submissions. Good luck to the <500 ACs and all the reviewers. God have mercy on the grad students who are just trying to graduate. I'm sure you're all doing good work even if the reviewer didn't read a word. I hope your pictures and tables are enough. [-1] https://twitter.com/CVPR/status/1722384482261508498 [0] https://www.jmlr.org/papers/v23/20-1433.html [1] https://web.njit.edu/~usman/courses/cs675_fall18/10.1.1.441.... [2] https://arxiv.org/abs/1606.08415 [3] https://www.cs.cmu.edu/~clean-fid/ [4] https://arxiv.org/abs/2010.11929 [5] https://arxiv.org/abs/2012.12877 [6] https://arxiv.org/abs/1512.03385
Original comment Simplifying Transformer Blocks
og_kalu
Neutral mention · rule
Vision Transformers don't use CNNs to extract anything first. (https://arxiv.org/abs/2010.11929). You could but it's not necessary and it doesn't add anything so it doesn't happen anymore. Vision transformers won't treat the image as a sequence of pixels but that's mostly because doing that gets very expensive very fast. The image is split into patches and the patches have positional embeddings.
Original comment What Is a Transformer Model?
mxkopy
Neutral mention · rule
> AI generates text - it does nothing other than that. It's admittedly great at manipulating text, but that's where it's "general purpose transforming" stops. I will refer you to the Vision Transformer: https://arxiv.org/abs/2010.11929 I suggest you do a literature review on the applications of ANNs, CNNs and attention mechanisms. They are quite numerous but I will concede that they generally 'just' do compression. AI encodes human concepts like emotions and spatial reasoning into vectors. Which, their analogous-ness to the real world, is more of a philosophical question than a practical one at this point. > It has impressive results, but comparing the value of human…
Read the full comment
labor to the work AI can do is startlingly parallel to comparing the value of a database to the work a blockchain can do. I mean 'human labor' is a very general thing, compared to 'database' IMO. > The first most common use case of AI is for military purposes. This is true but the AI that the military uses is probably a far cry from what the public is able to access - and not from a technological advancement but a specificity standpoint. The military will want deterministic, non-opaque methods that are probably closer to the policy-gradient recommendation systems of old. Think Netflix and TikTok, not GPT. I can say from an investor standpoint AI is probably a bubble but only because its entire foundations are open source. There's no money to be made from it, therefore it's a bubble. Crypto on the other hand always had somewhat dubious foundations. There are few use cases for it, therefore it's a bubble.
Original comment The AI hype bubble is the new crypto hype bubble
anshumankmr
Neutral mention · rule
>An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale >https://arxiv.org/abs/2010.11929 Very interesting. I didn't think of connecting these two dots. Let's see if it can be applied to other CV tasks, like object detection.
Original comment The Most Cited AI Papers in 2022
johnthewise
Neutral mention · rule
I don't quite remember whether it was first used in Vit paper[1], but it's a fairly straight forward idea. You take the patches of an image like they are words in a sentence, reduce the size of the patch(num_of_pixel x num_of_pixel) with a linear projection so that we can actually process it and get rid of sparse pixel information, add in positional encodings to put in location information of the patch and treat them as how you treated words in language models from that point on with transformers. Essentially, words are human constructed but information dense representation of language but images do have quite sparsity in them because individual pixel values don't really change much of an image. 1: https://arxiv.org/pdf/2010.11929.pdf
Original comment Imagen: An AI system that creates photorealistic images from input text
stevenwalton
Neutral mention · rule
Lots of people transfer learn with transformers. ViT[0] originally did CIFAR with it. Then DeiT[1] introduced some knowledge transfer (note: their student is larger than the teacher). ViT pretrained on both ImageNet21k and JFT-300m. CCT ([1] from above) was focused on training from scratch. There's two paradigms to be aware of. ImageNet and pre-training can often be beneficial but it doesn't always help. It really depends on the problem you're trying to tackle and if there are similar features within the target dataset and the pre-trained dataset. If there is low similarity you might as well train from scratch. Also, you might not want as large of models (like ViT and DeiT have, which ViT's has more parameters than CIFAR-10 has features). Disclosure: Author on CCT [0] https://arxiv.org/abs/2010.11929 [1] https://arxiv.org/abs/2012.12877
Original comment Transformers, originally designed to handle language, are taking on vision
marcyb5st
Neutral mention · rule
With very deep models based on Convolution (CNNs, the article is talking about AI on a computer vision problem and CNNs are the standard for these problems) you loose almost all explainability. There might be ways to partially mitigate that like adding additional "heads" to the model to output some debugging-like data, or run the image in patches and see where the model is more confident about a particular metric of interest. Both these options are not optimal as you still don't see the inner workings of the model in an interpretable way. Transformers models [1], on the other hand, are the de-facto standard for Natural Language Processing/Understanding problems and recent…
Read the full comment
studies [2] show that they can be applied to computer vision tasks. By construction, transformers let you access the so called "attention heads", which carry a good deal of information about what the model believes is important about the input. So that might be a promising route for some explainability in computer vision tasks, but we are quite a bit far away from fully understanding what is going on inside a deep learning model. [1] https://arxiv.org/abs/1706.03762 [2] https://arxiv.org/abs/2010.11929
Original comment Strangely Accurate AI Predictions from Blurry Medical Scans Alarms Researchers
unixpickle
Neutral mention · rule
I think you are conflating "Transformers" and "autoregressive models". Transformers are a general purpose architecture for transforming sequences into other sequences with self-attention. AR models / GANs are frameworks for generative modeling. The model architecture is almost entirely orthogonal to the generative framework. You can use transformers as part of GANs [1], and you can even use them as discriminative models for images [2]. [1] https://arxiv.org/abs/2102.07074 [2] https://arxiv.org/abs/2010.11929
Original comment Deep reinforcement learning is a waste of time (2019)
og_kalu
Neutral mention · rule
Just to add, the first user who replied to you is quite wrong. You can use CNNs to get features first...but it doesn't happen anymore. It's unnecessary and adds nothing. Pixels don't get fed into transformers but that's more expense than anything. Transformers need to understand how each piece relates to every other piece. That gets very costly very fast when the "pieces" are pixels. Images are split into patches instead with positional embeddings. as for how it learns the representations anyway. Well it's not like there's any specific intuition to it. after all, the original authors didn't anticipate the use case in Vision. and the fact that you didn't need CNNs to extract features first didn't really come into light till this paper - https://arxiv.org/abs/2010.11929 It basically just comes down to lots of layers and training
Original comment What Is a Transformer Model?

Keep following the thread.

More in Computer vision

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

Psychology
16 mentions14 months
2.5HN score