Bridging the 'reality gap' that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator. With enough variability in the simulator, the real world may appear to the model as just another variation. We focus on the task of object localization, which is a stepping stone to general robotic manipulation skills. We find that it is possible to train a real-world object detector that is accurate to $1.5$cm and robust to distractors and partial occlusions using only data from a simulator with non-realistic random textures. To demonstrate the capabilities of our detectors, we show they can be used to perform grasping in a cluttered environment. To our knowledge, this is the first successful transfer of a deep neural network trained only on simulated RGB images (without pre-training on real images) to the real world for the purpose of robotic control.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
This is important for vision ML because sufficiently advanced simulations may comprise the real world as a special case: https://arxiv.org/abs/1703.06907
For example, gaming pixel maps have been used for semantic simulation, and have been rendered scenes: https://arxiv.org/abs/1608.02192
https://www.cv-foundation.org/openaccess/content_cvpr_2016/p...
This concept has not (yet) been applied in audio ML. We have a paper in submission---will be on ArXiv soon---where we share a GPU-enabled modular synthesizer that is 16000x faster than realtime, concurrently released with a 1-billion audio sample corpus that is 100x larger than any audio dataset in the literature. Here's the code: https://github.com/torchsynth/torchsynth
what is your thought on synthetic data vs active learning?
For some domains, with privacy concerns or rarity of objects, getting labelled data for deep learning is challenging.
There is decent research on sim2real i.e transferring models trained on synthetic data to real world applications
https://arxiv.org/pdf/1703.06907.pdf
While the application is indeed a joke, the technique for sim-to-real transfer (and the robotic system in the post) is very real: https://arxiv.org/abs/1703.06907 :).
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.