Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, Cynthia Rudin
2020 · arXiv
The primary aim of single-image super-resolution is to construct high-resolution (HR) images from corresponding low-resolution (LR) inputs. In previous approaches, which have generally been supervised, the training objective typically measures a pixel-wise average distance between the super-resolved (SR) and HR images. Optimizing such metrics often leads to blurring, especially in high variance (detailed) regions. We propose an alternative formulation of the super-resolution problem based on creating realistic SR images that downscale correctly. We present an algorithm addressing this problem, PULSE (Photo Upsampling via Latent Space Exploration), which generates high-resolution, realistic images at resolutions previously unseen in the literature. It accomplishes this in an entirely self-supervised fashion and is not confined to a specific degradation operator used during training, unlike previous methods (which require supervised training on databases of LR-HR image pairs). Instead of starting with the LR image and slowly adding detail, PULSE traverses the high-resolution natural image manifold, searching for images that downscale to the original LR image. This is formalized through the "downscaling loss," which guides exploration through the latent space of a generative model. By leveraging properties of high-dimensional Gaussians, we restrict the search space to guarantee realistic outputs. PULSE thereby generates super-resolved images that both are realistic and downscale correctly. We show proof of concept of our approach in the domain of face super-resolution (i.e., face hallucination). We also present a discussion of the limitations and biases of the method as currently implemented with an accompanying model card with relevant metrics. Our method outperforms state-of-the-art methods in perceptual quality at higher resolutions and scale factors than previously possible.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
6 selected excerpts · 6 total mentions
It's from an algorithm called PULSE. It seems to search the latent space of StyleGAN to find a face that compresses to something similar to the query image. https://arxiv.org/pdf/2003.03808.pdf https://www.theverge.com/21298762/face-depixelizer-ai-machin...
That's because we're still bad about long-tailed data and that people outside the research don't realize that we're first prioritizing realistic images before we deal with long-tailed data (which is going to be the more generic form of bias). To be honest, it is a bit silly to focus on long-tailed data when results aren't great. That's why we see the constant pattern of getting good on a dataset and then focusing on the bias in that dataset. I mean a good example of this is the Pulse[0][1] paper. You may remember it as the white Obama. This became a huge debate and it was pretty easily shown that the largest factor was the dataset bias. This outrage did lead to fixing…
FFHQ but it also sparked a huge debate with LeCun (data centric bias) and Timnit (model centric bias) at the center. Though Pulse is still remembered for this bias, not for how they responded to it. I should also note that there is human bias in this case as we have a priori knowledge of what the upsampled image should look like (humans are pretty good at this when the small image is already recognizable but this is a difficult metric to mathematically calculate). It is fairly easy to find adversarial examples, where generative models produce biased results. It is FAR harder to fix these. Since this is known by the community but not by the public (and some community members focus on finding these holes but not fixing them) it creates outrage. Probably best for them to limit their release. [0] https://arxiv.org/abs/2003.03808 [1] https://cdn.vox-cdn.com/thumbor/MXX-mZqWLQZW8Fdx1ilcFEHR8Wk=...
The underlying (racially biased) superresolution algorithm was called PULSE and the code was released as an artifact of this paper: Paper: https://arxiv.org/pdf/2003.03808v1.pdf Here are the authors: Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, Cynthia Rudin They conveniently captured an image of them all in one place for us here - https://cdn.telanganatoday.com/wp-content/uploads/2020/06/Au... Pretty diverse team if you ask me. Sprinkling underrepresented (and therefore 'underpowered') representation into engineering teams will solve a lot of issues, but I think it has yet to be proven that it's going to help ML bias.
This animation gives a good overview of what it is doing: https://raw.githubusercontent.com/adamian98/pulse/master/rea... Basically it generates an upsampled image using a generative network, then downsamples it again to see how different it is from the original. It then tweaks the upsampled image (by moving around in the latent space of the generative model) until the downsampled version matches the original as closely as possible. Seems like it works really well, but only for faces. I wonder if the same technique could be applied to a broader domain of images? The actual paper can be found at https://arxiv.org/abs/2003.03808
I'm not sure why PULSE is being called out here. This is extremely common on any super-resolution trained on extremely biased datasets like FFHQ. In fact, this discussion is something that led to the dataset being changed. The authors actually extended their research to discuss the bias[0]. I'll also link to the Reddit discussion[1]. IMO the authors here responded to this correctly. It is important to remember that algorithms are only good on their in distribution datasets. And kudos on the authors for doing more experiments and including work on a less racially and sexually biased dataset. [0] section 6 https://arxiv.org/pdf/2003.03808.pdf [1] https://www.reddit.com/r/MachineLearning/comments/hk2ryn/d_h...
One of the underlying models, PULSE, was trained on CelebAHQ, which is likely what the results are mostly white-looking. StyleGAN, which was trained on the much more diverse (but sparse) FFHQ dataset does come up with a much more diverse set of faces[1]...but PULSE couldn't get them to converge very closely on the pixelated subjects...so they went with CelebA [2]. [1] https://github.com/NVlabs/stylegan [2] https://arxiv.org/pdf/2003.03808.pdf (ctrl+f ffhq)
More from the same topics.
Tim Kraska, Alex Beutel, Ed H. Chi +2
Indexes are models: a B-Tree-Index can be seen as a model to map a key to the position of a record within a sorted array, a Hash-Index as a model to map a key to a position of a record within an unsorted array, and a BitMap-Index as a model to indicate if a data record exists or not.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Martin Kleppmann
The CAP Theorem is a frequently cited impossibility result in distributed systems, especially among NoSQL distributed databases.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.