Oran Gafni, Adam Polyak, Oron Ashual
Shelly Sheynin, Devi Parikh, Yaniv Taigman
2022 · arXiv
Recent text-to-image generation methods provide a simple yet exciting conversion capability between text and image domains. While these methods have incrementally improved the generated image fidelity and text relevancy, several pivotal gaps remain unanswered, limiting applicability and quality. We propose a novel text-to-image method that addresses these gaps by (i) enabling a simple control mechanism complementary to text in the form of a scene, (ii) introducing elements that substantially improve the tokenization process by employing domain-specific knowledge over key image regions (faces and salient objects), and (iii) adapting classifier-free guidance for the transformer use case. Our model achieves state-of-the-art FID and human evaluation results, unlocking the ability to generate high fidelity images in a resolution of 512x512 pixels, significantly improving visual quality. Through scene controllability, we introduce several new capabilities: (i) Scene editing, (ii) text editing with anchor scenes, (iii) overcoming out-of-distribution text prompts, and (iv) story illustration generation, as demonstrated in the story we wrote.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
6 selected excerpts · 6 total mentions
More examples here https://www.instagram.com/p/Cf_tjL_rgEe/?hl=en I understand this is probably just a frontend to this paper from March https://arxiv.org/pdf/2203.13131.pdf Given the spotlight DALL-E has received makes sense to capitalise
Facebook's Make-A-Scene, which looks almost as good as DALL-E 2, does something very much like that, using face-specific targeting to boost the quality of that part of images: https://arxiv.org/abs/2203.13131#facebook
> What is the content of the training set like, that DALL·E 2 has all this info? Web crawling. (https://arxiv.org/pdf/2103.00020.pdf 2.2) > Are there people out there spending thousands of hours just tagging or writing descriptions of training images? "Yes", Google Image Search, Wikipedia captions, Danbooru are all that in some sense. The biases should be obvious if you think about it for a minute and CLIP has even more biases since they removed all kinds of NSFW content, etc. > And does DALL·E just receive the image as-is with its description, or does it get more info (e.g. "this part of the image is the dog")? Only that, but segmentation is a big research area too and it looks like FB has a reproduction with some of that mixed in. https://arxiv.org/abs/2203.13131
Also Make-A-Scene, which in some ways is noticeably better than DALL-E 2 (faces, editing & control of layout through semantic segmentation conditioning): https://arxiv.org/abs/2203.13131#facebook
It's the PR risk. You can easily pass the 'look good to an uniformed observer' test with human faces. Remember, faces were something GANs were doing near-flawlessly back as far as ProGAN all the way back in the dark ages of late 2017. (Then StyleGAN did non-photograph faces, like anime - see my ThisWaifuDoesNotExist for a demo of that.) Doing them as part of a larger composition, where the face is a small part of the images and may vary much more than in closeup centered portraits, is harder, but take a look at how Facebook's DALL-E rival Make-A-Scene does it: https://arxiv.org/abs/2203.13131#facebook They specially target faces as part of the training process, with face-specific detectors/losses, and so the faces come out great.
As I keep telling people: "hands are hard". This is why I went so far as to make a hand-specific dataset ("PALM" https://www.gwern.net/Crops#palm which of course now everyone is going to confuse with 'PaLM'...). Hands are just way too variable to learn easily. My dataset is a start, but it may benefit from focused training, the way Facebook's new Make-A-Scene https://arxiv.org/abs/2203.13131#facebook (not DALL-E 2 quality but not far from it) has focused losses on faces.
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.