Tero Karras, Samuli Laine, Miika Aittala
Janne Hellsten, Jaakko Lehtinen, Timo Aila
2019 · arXiv
The style-based GAN architecture (StyleGAN) yields state-of-the-art results in data-driven unconditional generative image modeling. We expose and analyze several of its characteristic artifacts, and propose changes in both model architecture and training methods to address them. In particular, we redesign the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images. In addition to improving image quality, this path length regularizer yields the additional benefit that the generator becomes significantly easier to invert. This makes it possible to reliably attribute a generated image to a particular network. We furthermore visualize how well the generator utilizes its output resolution, and identify a capacity problem, motivating us to train larger models for additional quality improvements. Overall, our improved model redefines the state of the art in unconditional image modeling, both in terms of existing distribution quality metrics as well as perceived image quality.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
5 selected excerpts · 5 total mentions
I've been working a lot on my blog, which entails training and implementing neural networks. My next two posts are going to be about: - https://arxiv.org/pdf/1912.04958 - StyleGAN2 - https://arxiv.org/pdf/1807.09341 - Causal InfoGAN, written in TinyGrad https://ym2132.github.io
> The denoising part of a denoising autoencoder refers to the noise applied to its input Agree, it converts a noisy image to a denoised image. But the odd thing is, when you put a noisy image into a StyleGAN2 encoder, you get latents which the decoder will turn into a de-noised image. So in practical use, you can take a trained StyleGAN2 encoder/decoder pair and use it as if it was a denoiser. For example https://arxiv.org/abs/2103.04192 > These differences lead to learned distributions in the latent space that are entirely different I also agree there. The training for a denoising auto-encoder and for a GAN network is different, leading to different distributions which…
are sampled for generating the images. But the architecture is still very similar, meaning the limits of what can be learned should be the same. > Beyond that the comparison just doesn't work, yes there are two networks but the discriminator doesn't play the role of the AE's encoder at all Yes, the discriminator in a GAN won't work like an encoder. But if you look at how StyleGAN 1/2 are used in practice, people combine it with a so-called "projection", which is effectively an encoder to convert images to latents. So people use a pipeline of "image to latent encoder" + "latent to image decoder". That whole pipeline is very similar to an auto-encoder. For example, here's an NVIDIA paper about how they round-trip from image to latent to image with StyleGAN: https://arxiv.org/abs/1912.04958 My interpretation of what they did in that paper is that they effectively trained a StyleGAN-like model with the image L2 loss typically used for training a denoising auto-encoder.
To be honest I don't think the StyleGAN papers are benchmark chasing. If you read StyleGAN[0], StyleGAN2[1], StyleGAN2-ADA[2], and this paper there is a clear story. They call out the mistakes in the previous papers and resolve them. The papers themselves even admit to where they fall short. But it is research. Problems don't get solved all at once. But if you pay attention to these 4 papers it is very clear Kerras has a very well defined research focus and direction. He's showing his progress over time, sharing it with the community, and learning from the community as well. This is how research should happen. [0] https://arxiv.org/abs/1812.04948 [1] https://arxiv.org/abs/1912.04958 [2] https://arxiv.org/abs/2006.06676
Seems like it has been vastly improved since then [1]. The results are much better than when originally posted in HN. [1] - https://arxiv.org/abs/1912.04958
The StyleGAN2 paper[0] actually addresses some of the biggest clues. They didn't completely solve everything but symmetry is one of the biggest indicators. Specifically ears and eyes. But also really pay attention to the teeth. If you go to ThisPersonDoesNotExist[1] you'll see that there's still phase issues with teeth and eyes. It's an improvement on the original paper but it is still weird. For eyes you'll notice things like them not pointing in the same direction (this isn't always super obvious). Another big thing to look at is the neck. Sometimes women will get Adam's Apples and men will lack them. Collars won't be symmetric and necks can have weird wrinkles. Lastly…
pay close attention to the background because that's where you'll likely find monsters. Lots of portrait photos have blurred background so "lack of detail" isn't a great metric. But backgrounds that are a bit disjoint are great indicators. None of these metrics alone is a great tool and you gotta use them together (unless someone messed up big time when selecting the photo). Poke around through [1] a bit and you'll start seeing some of the features I'm discussing. But I should also note that these flaws are known and being worked on so this information's usefulness will degrade with time (it's still good information because it helps us know how to look and what's difficult about faces). These of course go along with standard identification techniques like lighting, shadows, etc. [0] https://arxiv.org/abs/1912.04958 [1] https://thispersondoesnotexist.com/
More from the same topics.
Dan Tong, David J. Farnham, Lei Duan +4
Abstract If future net-zero emissions energy systems rely heavily on solar and wind resources, spatial and temporal mismatches between resource availability and electricity demand may challenge system reliability.
Steven M. Paul, Daniel S. Mytelka, Christopher T. Dunwiddie +4