Skip to content
The methodOpen data
Back to the collection

Learning from Simulated and Unsupervised Images through Adversarial Training

Ashish Shrivastava, Tomas Pfister, Oncel Tuzel

Show all 6 authors

Josh Susskind, Wenda Wang, Russ Webb

2016 · arXiv

Algorithms
Read the paper
HN score
0
Mentions
7
Readers
7
Months
4
What’s in a score?

From the abstract

With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulator's output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications to the standard GAN algorithm to preserve annotations, avoid artifacts, and stabilize training: (i) a 'self-regularization' term, (ii) a local adversarial loss, and (iii) updating the discriminator using a history of refined images. We show that this enables generation of highly realistic images, which we demonstrate both qualitatively and with a user study. We quantitatively evaluate the generated images by training models for gaze estimation and hand pose estimation. We show a significant improvement over using synthetic images, and achieve state-of-the-art results on the MPIIGaze dataset without any labeled real data.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

7 selected excerpts · 7 total mentions

dimatura
Neutral mention · rule
A bit of a tangent, but reminds me of this article, RenderGAN [1]. The main point of the article was about using 3D rendering + GAN to generate convincing synthetic data, and their application was creating training data to detect "barcodes" [2] attached to bees. It's a similar premise and method than Apple's SUGAN paper [3], that came out around the same time, to much more fanfare (it won CVPR best paper award iirc). I think the impact of renderGAN was a bit limited by the narrow scope of their application (even narrower than Apple's eye tracker application) and the fact that the generator had some handcrafted layers that didn't really generalize well outside the black…
Read the full comment
and white marker application. [1] https://www.frontiersin.org/articles/10.3389/frobt.2018.0006... [2] Technically not a barcode since it doesn't have bars, it's a circular 2D marker, but close enough. For that matter the QR-like codes aren't really "bar"codes either, but that's just pedantry. [3] https://arxiv.org/abs/1612.07828
Original comment Barcoding Bees (2020)
rgovostes
Neutral mention · rule
Apple's first published ML paper[1] described how they trained a neural network to estimate eye gaze. To augment the limited data set of labeled eye photos, they rendered synthetic eyes and then used adversarial training to make them match the real-life data set. This paper came out several months before the iPhone X introduced Face ID with eye gaze tracking, so it can be seen as a hint of things that were to come. What's intriguing to me is that the paper also talks about doing the same with synthetic depth maps of hands, adding details such as occlusion and imperfectly-captured edges. With Apple investing heavily in ARKit, and rumors of an AR headset, it seems like hand tracking is inevitable. 1: https://arxiv.org/abs/1612.07828
Original comment Oculus: Hand Tracking Updates
alexbeloi
Neutral mention · rule
I spoke in hyperbole, there were dozens of papers using generative networks to create real looking simulated data which is the more common approach. You start with noise + some features you want in your image, then you turn that into a realistic image. The notable exception is if you want to generate segmentation data (where you want labels for each pixel), in this case the set of features you want is essentially an image already. ICCV 2017 * Photographic Image Synthesis with Cascaded Refinement Networks: https://arxiv.org/pdf/1707.09405.pdf * Synthetic avatars: https://arxiv.org/pdf/1704.05693.pdf * Synthesizing cell microscopy data:…
Read the full comment
https://arxiv.org/pdf/1708.04692.pdf * Material Editing Using a Physically Based Rendering Network: https://arxiv.org/pdf/1708.00106.pdf * 3D-PRNN: Generating Shape Primitives With Recurrent Neural Networks: * for more search 'generative' 'generating' 'simulating' 'synthesis' 'synthetic' on: http://iccv2017.thecvf.com/program/main_conference#schedule Also worth mentioning: CVPR 2017 best paper, this is the other exception, it's unique because it doesn't require _any_ labeled data * Learning from Simulated and Unsupervised Images through Adversarial Training: https://arxiv.org/abs/1612.07828
Original comment Closing the Simulation-to-Reality Gap for Deep Robotic Learning
iraphael
Neutral mention · rule
* Densely Connected Convolutional Networks [0] by Gao Huang, Zhuang Liu, Laurens van der Maaten, & Kilian Q. Weinberger. * Learning from Simulated and Unsupervised Images through Adversarial Training [1] by Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, & Russell Webb CVPR 2017 Best Paper Honorable Mention Awards * Annotating Object Instances with a Polygon-RNN [2] by Lluís Castrejón, Kaustav Kundu, Raquel Urtasun, & Sanja Fidler * YOLO9000: Better, Faster, Stronger [3] by Joseph Redmon & Ali Farhadi CVPR 2017 Best Student Paper Award * Computational Imaging on the Electric Grid [4] by Mark Sheinin, Yoav Y. Schechner, & Kiriakos N.…
Read the full comment
Kutulakos CVPR 2017 Longuet-Higgins Prize (for cvpr 2007 papers that stood the test of time) * Object Retrieval with Large Vocabularies and Fast Spatial Matching [5] by James Philbin, Ondrej Chum, Michael Isard, Josef Sivic & Andrew Zisserman CVPR 2017 PAMI Young Researcher Award * Ross Girshick & Julien Mairal [0] https://arxiv.org/abs/1608.06993 [1] https://arxiv.org/abs/1612.07828 [2] https://arxiv.org/abs/1704.05548 [3] https://arxiv.org/abs/1612.08242 [4] http://openaccess.thecvf.com/content_cvpr_2017/papers/Sheini... [5] http://ieeexplore.ieee.org/document/4270197/
Original comment CVPR 2017 Best Paper Awards
ssivark
Neutral mention · rule
It is not an anonymous publication! 1. The article is based on a paper submitted to the prestigious CVPR, which is obviously not anonymous. ( https://arxiv.org/abs/1612.07828 ) They mention this at the end of the post. 2. The blog's "journal" name is misleading. It means "journal" in the sense of a record of some of the stuff happening at Apple. My guess is that the blog post does not name authors because it was written by an Apple representative who manages public communications---probably because Apple is very very particular about managing their brand and what they say publicly. I wouldn't put it beyond Apple to hire someone well-versed in ML just for this role. So,…
Read the full comment
while they are starting to allow researchers to publish, they have quite some distance to go to encourage their researchers to communicate freely. One step at a time, I guess... PS: Somehow the SNAFU of deliberately calling it a "journal" is very reminiscent of Steve Jobs's chutzpah.
Original comment Apple Machine Learning Journal
zo7
Neutral mention · rule
It's interesting how much criticism they're getting because Apple formatted their blog to be anonymous and watered down, but they're clear in their first technical post that it is just an overview of work that the researchers are presenting at CVPR [1]. So the researchers at Apple are still getting credit for their work in the scientific community, but the PR-facing side of their work is anonymous, probably for some aesthetic reason (this is Apple, of course) [1]: https://arxiv.org/abs/1612.07828
Original comment Apple Machine Learning Journal
Stasis5001
Neutral mention · rule
A lot of academic papers actually aren't all that great, for a variety of reasons. Normally, you can use citations, journal, and author credentials to get a sense of whether a paper is even worth skimming. The only "paper" on the "journal" right now looks like it's just a watered-down, html-only version of https://arxiv.org/abs/1612.07828! Seems like more of a PR stunt than anything useful, but who knows.
Original comment Apple Machine Learning Journal

Keep following the thread.

More in Algorithms

More from the same topics.

2000arXiv

Dancing links

Donald E. Knuth

The author presents two tricks to accelerate depth-first search algorithms for a class of combinatorial puzzle problems, such as tiling a tray by a fixed set of polyominoes.

AlgorithmsProgramming languages
10 mentions10 months
2HN score