Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer
2021 · arXiv
By decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining. However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations. To enable DM training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pretrained autoencoders. In contrast to previous work, training diffusion models on such a representation allows for the first time to reach a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity. By introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes and high-resolution synthesis becomes possible in a convolutional manner. Our latent diffusion models (LDMs) achieve a new state of the art for image inpainting and highly competitive performance on various tasks, including unconditional image generation, semantic scene synthesis, and super-resolution, while significantly reducing computational requirements compared to pixel-based DMs. Code is available at https://github.com/CompVis/latent-diffusion .
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
9 selected excerpts · 9 total mentions
> But no LLM used for that? Well, I mean, I self-hosted stable diffusion to be able to generate the teaser images and the award images (for the ranks in the Game Over Dialog). I only implemented the initial SD paper [1] back then so I have no clue whether the u-net architecture changed when it comes to the semantic mapping of the text/labels. Did that change and is Stable Diffusion 3 now an LLM, actually? [1] https://arxiv.org/pdf/2112.10752
> I don't think the "model collapse" problem is particularly important these days. I think you might misunderstand what model collapse is. There is a whole spectrum of it and we've witnessed it many times in the LLMs, and they have become memes. A fairly recent example is the Golden Gate Claude[0]. This is mode{,l} collapse. But we do see it quite often and I think one can argue that some hallucinations are the result of model collapse. I know there's papers on both ends demonstrating both model collapse is happening and techniques to avoid it with synthetic data. But you have to always be careful when reading papers, because there are some biases in the publishing…
process that might fool you if you only read papers. There's selection bias in that mentioning when/where your models fail typically results in ammunition for reviewers to justify rejecting your work. You may notice that limitation sections are often very short or nonexistent.[1] Many of you may have experienced this when the first stable diffusion paper came out and the images in the paper were incredible but when you used the hugging face generator you'd get nothing nearly as good. Hell, try even now[2]. Can you do better than I did? Sure! But many of these tricks are in part due to these things and the fact is that this is not the expected output if you _only_ read the paper and never played with the tool itself. That there's a big difference between these. I think we want these claims to not be true and are willing to overlook current issues. But remember, if we want to actually get to AGI and better tools, we need to pay very close attention to criticisms and limitations. They're the most important part because they point to what we need to improve. Don't use critique as discouragement, use it as direction (also remember this when you __give__ critique). [0] https://news.ycombinator.com/item?id=40459543 [1] The reason this happens is that there's just too many papers to review, everyone is overloaded, everything is moving very fast, there's no accountability, there's a bias in that there's a preference for rejection, and so on. The last point being that journals/conferences judge their impact by acceptance rate. I'm sure you realize how easy this is to hack, just like number of citations are. Especially when there's tons of money involved like in ML. [2] https://imgur.com/a/xscyp1X using https://huggingface.co/spaces/stabilityai/stable-diffusion-3... Stability's page: https://stability.ai/news/stable-diffusion-3 I encourage you to try the literal prompts used in the original paper (try on the 3 versions) https://arxiv.org/abs/2112.10752
Everything you've said is _intuitively_ correct, but empirically wrong. I've experimented with training VAEs for audio diffusion for the last few months and here's what I found: - Although the best results for a stand-alone VAE might require increasing the KL loss weight as high as you can to reach an isotropic gaussian latent space without compromising reconstruction quality, beyond a certain point this actually substantially decreases the ability of the diffusion model to properly interpret the latent space and degrades generation quality. The motivation behind constraining the KL loss weight is to ensure the VAE only provides _perceptual_ compression, which VAEs are…
quite good at, not _semantic_ compression, for which VAEs are a poor generative model compared to diffusion. This is explained in the original latent diffusion paper on which Stable Diffusion was based: https://arxiv.org/pdf/2112.10752.pdf - You're correct that trading dimensions for channels is a very easy way to increase reconstruction quality of a stand-alone VAE, but it is a very poor choice when the latents are going into a diffusion model. This again makes the latent space harder for the diffusion model to interpret, and again isn't needed if the VAE is strictly operating in the perceptual compression regime as opposed to the semantic compression regime. The underlying reason is channel-wise degrees of freedom have no inherent structure imposed by the underlying convolutional network; in the limit where you hypothetically compress dimensions to a single point with a large number of channels the latent space is completely unstructured and the entropy of the latents is fully maximized; there are no patterns left whatsoever for the diffusion model to work with. TLDR: Designing VAEs for latent diffusion has a different set of design constraints than designing a VAE as a stand-alone generative model.
The model was initially released in August 22, 2022, and everything from the GitHub organization that hosted the code, to the actual licenses being around and all the communication made it very clear that there was a bunch of groups involved with the model; Runway, CompVis, and Stability AI Then Stability AI raised money from investors in October 2022. Are you really telling me these investors didn't even do any sort of due diligence (not even "barely any" but literally any at all), didn't realized that Stable Diffusion was the work of many, and checked the license of the model and code? That's a bit far fetched, to be honest. Supposedly, this "leaked pitch deck" is…
supposed to show us that they lied to investors, but where is the pitch deck itself? Seemingly, only one of the slides is in the article, how could anyone reach any sort of conclusion based on just one slide? Edit: The article seems to be some sort of clickbait trash that is so rampant around the web... One selected part: > The Stable Diffusion code was released by researchers at LMU Munich in April 2022. Links to https://arxiv.org/abs/2112.10752 which is the Version 2 of the paper, indeed released in April 2022. However, that paper is about, and links to https://github.com/CompVis/latent-diffusion which is not Stable Diffusion, it's Latent Diffusion. Stable Diffusion was released in August 2022, and is at https://github.com/CompVis/stable-diffusion
Very interesting, although I was expecting the Stable Diffusion paper[1] to be one of the most cited in 2022 [1]: High-Resolution Image Synthesis With Latent Diffusion Models, https://arxiv.org/abs/2112.10752
Yes, and another reason for the small model size and the novelty of the underlying paper [1], is that the diffusion model is not acting on the pixel space but rather on a latent space. This means that this 'latent diffusion model' does not only learn the task at hand (image synthesis) but in parallel also a powerful lossy compression model via an outer auto encoder structure. Now, the number of weights (model size) can be reduced drastically as the inner neural network layers act on a lower dimensional latent space rather than a high dimensional pixel space. It's fascinating because it shows that deep learning at its core comes down to compression/decompression (encoding/decoding), with close relation to Shannon's Information Theory (e.g. source coding/channel coding/data processing inequality). [1] https://arxiv.org/abs/2112.10752
It is fed using a fascinating mechanism called "cross-attention" that originated in the NN architecture called Transformer - which was used to achieve state-of-the-art translations. It uses something like associative memory, where a NN inside Stable Diffusion, that generates image (UNet, working in latent space), at almost each step "asks" the whole encoded prompt to provide data at various positions using query vector Q that is matched against key vectors K and value vectors V [0]. How Stable Diffusion works [1] as a whole is not really hard to comprehend at a high level - but you'll need some prereqs - probability theory underlying this is explained in Variational…
Autoencoders [2], then Diffusion Models [3] sort of made a really cool "deep variational" autoencoder that uses small noise-denoise steps, but largely the same math (variational inference), but they were unwieldy because operated in pixel space, after that Latent Diffusion Models [4] democratized the thing by vastly reducing the amount of computation needed - operating in latent space (btw that's why the images in this HN post look so cool - the denoising is not in the pixel space!). [0] https://jalammar.github.io/illustrated-transformer/ [1] https://huggingface.co/blog/stable_diffusion [2] https://arxiv.org/abs/1906.02691 [3] https://arxiv.org/abs/2006.11239 [4] https://arxiv.org/abs/2112.10752
It is from the Heidelberg giants, Patrick Esser and Robin Rombach They continued building off their latent diffusion direction (encode with vqgan-vae and then diffusion in latent space) All roads lead to rome https://github.com/CompVis/stable-diffusion https://arxiv.org/abs/2112.10752
Yeah no problem, this is even closer to my area of focus! What do you know about physics and thermodynamics? I'd say a good intro for low background is from Tomczak[0]. He has a book, but the blog posts are nearly identical. He did a post doc with Max Welling (someone you should learn about if you want to get deep, like I was suggesting before). So I'd switch things up slightly. I'd go Intro -> Autoregressive -> Flow -> VAE -> Hierarchical VAE -> Energy Based Models -> Diffusion. It is worth learning about GANs btw, but this progression should be natural and build up. Continuing from there, you're going to want to learn about things Langevin Dynamics, Score Matching, and…
so on. Start with Yang Song's blogs[1]. Your goal should be to understand this paper[2]. Once you get there, you should be able to understand the famous DDPM paper[3]. But why we went through Tomczak wasn't just to get a good understanding of diffusion at a deeper level, but because you need these tools to understand Stable Diffusion which really is just Latent Diffusion[4]. This should connect back with Tomczak's 2 Improving VAE papers and you should also be able to understand NVAE. This is probably the quickest way to get you to a good understanding but if you want to dig deeper, which I highly encourage (because there are major issues that people aren't discussing) then you'll need more time. But you'll probably have to tools to do so if you go through this route. Other people I suggest looking into: Diederik Kingma, Ruiqi Gao, Stefano Ermon, Jonathan Ho, Ricky T. Q. Chen, and Arash Vahdat. [0] https://jmtomczak.github.io/ [1] https://yang-song.net/ [2] Deep Unsupervised Learning using Nonequilibrium Thermodynamics https://arxiv.org/abs/1503.03585 [3] https://arxiv.org/abs/2006.11239 [4] High-Resolution Image Synthesis with Latent Diffusion Models https://arxiv.org/abs/2112.10752
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Scott Aaronson
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.