Skip to content
The methodOpen data
Back to the collection

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Chitwan Saharia, William Chan, Saurabh Saxena

Show all 14 authors

Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, Mohammad Norouzi

2022 · arXiv

PhysicsMachine learningLanguage models
Read the paper
HN score
0
Mentions
5
Readers
5
Months
5
What’s in a score?

From the abstract

We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-fidelity image generation. Our key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model. Imagen achieves a new state-of-the-art FID score of 7.27 on the COCO dataset, without ever training on COCO, and human raters find Imagen samples to be on par with the COCO data itself in image-text alignment. To assess text-to-image models in greater depth, we introduce DrawBench, a comprehensive and challenging benchmark for text-to-image models. With DrawBench, we compare Imagen with recent methods including VQ-GAN+CLIP, Latent Diffusion Models, and DALL-E 2, and find that human raters prefer Imagen over other models in side-by-side comparisons, both in terms of sample quality and image-text alignment. See https://imagen.research.google/ for an overview of the results.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

5 selected excerpts · 5 total mentions

airgapstopgap
Neutral mention · rule
Diffusion is more parameter-efficient and you quickly saturate the target fidelity, especially with some refiner cascade. It's a solved problem. You do not need more than maybe 4B total. Images are far more redundant than text. In fact, most interesting papers since Imagen show that you get more mileage out of scaling the text encoder part, which is, of course, a Transformer. This is what drives accuracy, text rendering, compositionality, parsing edge cases. In SD 1.5 the text encoder part (CLIP ViT-L/14) takes a measly 123M parameters.[1] In Imagen, it was T5-XXL with 4.6B [2]. I am interested in someone trying to use a really strong encoder baseline – maybe from a…
Read the full comment
UL2-20B – to push this tactic further. Seeing as you can throw out diffusion altogether and synthesize images with transformers [3], there is no reason to prioritize the diffusion part as such. 1. https://forums.fast.ai/t/stable-diffusion-parameter-budget-a... 2. https://arxiv.org/abs/2205.11487 3. https://arxiv.org/abs/2301.00704
Original comment Stability AI releases its latest image-generating model, Stable Diffusion XL 1.0
godelski
Neutral mention · rule
I definitely don't buy these papers at face value. I say this as an ML researcher btw. You'll often see these works discussing zero-shot performance. But many of these tasks are either not zero-shot or even a known n-shot. Let's take a good example, Imagen[0] claims zero-shot MS-COCO performance but trains on LAION. COCO classes exist in LAION and there are similar texts. Explore COCO[1] and explore clip retrieval[2] for LAION. The example given is the first sample from COCO aircraft and you'll find almost identical images and captions with many of the same keywords. This isn't zero-shot. Why's this matter? Dataset contamination[3] being used in the evaluation process.…
Read the full comment
You can't conclude that a model has learned something if it has access to the evaluation data. Test sets have always been a proxy for generalization and MUST be recognized as proxies. This gets really difficult with LLMs where all we know is that they've scrapped a large swath of the internet and that includes GitHub and Reddit. I show some explicit examples and explanation with code generation here [4]. From there you might even see how it is difficult to generate novel test sets that aren't actually contaminated, which is my complaint about HumanEval. I show that we can find dupes or near dupes on GitHub despite these being "hand written." As per your sources all use GPT, which we don't know what data they have and don't have. But we do know they were trained on Reddit and GitHub. That should be enough to tell you that certain things like Physics and Coding problems[5] are spoiled. If you look at all the datasets used for evaluation in the works you listed I think you'll find reason to believe that there's a good chance that these too are spoiled. (Other datasets are spoiled and there's lots of experimentation that demonstrates the causal reasoning isn't as good as the performance suggests) Now mind you, this doesn't mean that LMs can't do causal reasoning. They definite can. Including causal discovery[6]. But this all tells us that it is fucking hard to evaluate models and even harder when we don't know what they were trained on. That maybe we need to be a bit more nuanced and stop claiming things so confidently. There's a lot of people trying to sell snake oil right now. These are very powerful tools that are going to change the world, but they are complex and people don't know much about them. We saw many snake oil salesmen at the birth of the internet too. Didn't mean the internet wasn't important or not going to change the course of humanity. Just meant that people were profiting off of the confusion and complexity. [0] https://arxiv.org/abs/2205.11487 [1] https://cocodataset.org/#explore [2] https://rom1504.github.io/clip-retrieval/?back=https%3A%2F%2.... [3] https://twitter.com/alon_jacovi/status/1659212730300268544 [4] https://news.ycombinator.com/item?id=35806152 [5] https://twitter.com/random_walker/status/1637929631037927424 [6] https://arxiv.org/abs/2011.02268
Original comment Governance of Superintelligence
espadrine
Neutral mention · rule
Model sizes are adjusted to reach a given quality. Many papers justified empirically the use of more neurons allocated to language than to visual understanding, like Imagen[0] (figure 4.a). as it yields better results. Debatably, it is also true in the human brain. To me, it could highlight that there is more entropy in language than in images. After all, games have a similar property: some games, like noughts and crosses, or checkers, can have the same model architecture reach optimal play with smaller sizes than are necessary for chess and go[1]. It is certainly true that language has a lot of rules (grammatical, vocabular, syntactic) which are necessary to master it,…
Read the full comment
but irrelevant to having a mental model of a scene; it just sequences a state of things into a stream of symbols. Maybe an additional entropy to learn in language which is absent in images includes motion and emotion. [0]: https://arxiv.org/pdf/2205.11487.pdf [1]: https://arxiv.org/pdf/2104.03113.pdf
Original comment Visual ChatGPT
nodja
Neutral mention · rule
They made T5 and released the weights, while T5 "only" goes up to 11B params, it's an efficient model for a wide variety of tasks and the base for a lot of other models/projects. Of note imagen[1] and open-assistant[2] use T5 and FlanT5. [1] https://arxiv.org/abs/2205.11487 [2] https://github.com/LAION-AI/Open-Assistant
Original comment Google Invests Almost $400M in ChatGPT Rival Anthropic

Keep following the thread.

More in Physics

More from the same topics.