Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio
2016 · arXiv
Unsupervised learning of probabilistic models is a central yet challenging problem in machine learning. Specifically, designing models with tractable learning, sampling, inference and evaluation is crucial in solving this task. We extend the space of such models using real-valued non-volume preserving (real NVP) transformations, a set of powerful invertible and learnable transformations, resulting in an unsupervised learning algorithm with exact log-likelihood computation, exact sampling, exact inference of latent variables, and an interpretable latent space. We demonstrate its ability to model natural images on four datasets through sampling, log-likelihood evaluation and latent variable manipulations.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
4 selected excerpts · 4 total mentions
Another example is RealNVP, which has the same structure to enable inversion (partition, process half, then use the result to mutate the other half in an invertible way). Sadly, I gather the computational performance Is pretty bad .. https://arxiv.org/abs/1605.08803
Nice work. I can think of only two things that are missing: * Normalizing flows - e.g., https://arxiv.org/abs/1605.08803 , https://arxiv.org/abs/1807.03039 , among many others * ODEnets and continous normalizing flows - https://arxiv.org/abs/1806.07366
Yes, although the particular technique used here (variational autoencoder, or VAE) doesn't benefit from increasing the code space due to a particular penalty (KL divergence against a fixed N(0, 1) Gaussian prior) during training. So even by making it go to 1000 digits in the code space, the model will probably still only choose to use 5 or 10 dimensions unless the KL penalty is relaxed, making it closer to a standard autoencoder. An autoencoder with a much larger code space could be an improvement, or some newer works such as Adversarially Learned Inference / Adversarial Feature Learning [0, 1] or real NVP [2, 3] could probably do a much better job at the task, at the…
cost of increased computation. Also something like inpainting parts of the frames with pixel RNN [4] would be interesting. [0] http://arxiv.org/abs/1606.00704 [1] http://arxiv.org/abs/1605.09782 [2] http://www-etud.iro.umontreal.ca/~dinhlaur/real_nvp_visual/ [3] http://arxiv.org/abs/1605.08803 [4] http://arxiv.org/abs/1601.06759
These guys have done what looks like an impressive amount of work... but I was disappointed to see there's no mention of generative language modeling at all and only a very brief mention of key advances in image generation over the past few years. Examples of generative language modeling ignored by this paper (not even mentioned): * Unsupervised Transformer - https://blog.openai.com/language-unsupervised/ * ELMo - https://arxiv.org/abs/1802.05365 * ULMFit - https://arxiv.org/abs/1801.06146 Examples of generative image modeling ignored by this paper (some are mentioned only in passing): * Glow - https://blog.openai.com/glow/ * RealNVP - https://arxiv.org/abs/1605.08803 * NICE - https://arxiv.org/abs/1410.8516 * Pixel CNN - https://arxiv.org/abs/1606.05328 (and its cousin PixelRNN - https://arxiv.org/abs/1601.06759)
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Haoran Xu, Fredrik Kjolstad
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.