Aditya Ramesh, Mikhail Pavlov, Gabriel Goh
Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, Ilya Sutskever
2021 · arXiv
Text-to-image generation has traditionally focused on finding better modeling assumptions for training on a fixed dataset. These assumptions might involve complex architectures, auxiliary losses, or side information such as object part labels or segmentation masks supplied during training. We describe a simple approach for this task based on a transformer that autoregressively models the text and image tokens as a single stream of data. With sufficient data and scale, our approach is competitive with previous domain-specific models when evaluated in a zero-shot fashion.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
4 selected excerpts · 4 total mentions
DALL-E 1 used a GPT approach to generate both text and images. Images are divided into patches, about 1024 patches for one image. Each patch is like a text token. > We describe a simple approach for this task based on a transformer that autoregressively models the text and image tokens as a single stream of data. https://arxiv.org/abs/2102.12092 The moral - you can just stream together text and images into a GPT style model.
If they use BPE dropout, then the split can be different and not unique. And for the record, they use BPE dropout for DALLE-1, see https://arxiv.org/pdf/2102.12092.pdf
Their research remains open: https://arxiv.org/abs/2102.12092 Their code/models are indeed closed, but there is no realistic alternative. If they let the public have unrestricted access, deepfakes + child images would appear on Day 1, and OpenAI would get cancelled. For OpenAI to survive, it has to be closed source.
DALL-E is 250m: https://arxiv.org/pdf/2102.12092.pdf#page=14 (so 20x larger). CLIP (https://cdn.openai.com/papers/Learning_Transferable_Visual_M...) is 400m (34x larger). And ALIGN (https://arxiv.org/abs/2102.05918#google) is 1800m (155x). (For 47m maybe you're thinking of a cleaned subset from YFCC100M or something that one of them used, but they all use total images an order or two larger than Wit.)
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.