Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra
2015 · arXiv
This paper introduces the Deep Recurrent Attentive Writer (DRAW) neural network architecture for image generation. DRAW networks combine a novel spatial attention mechanism that mimics the foveation of the human eye, with a sequential variational auto-encoding framework that allows for the iterative construction of complex images. The system substantially improves on the state of the art for generative models on MNIST, and, when trained on the Street View House Numbers dataset, it generates images that cannot be distinguished from real data with the naked eye.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
3 selected excerpts · 3 total mentions
> But much more interesting to me is a lossless domain-specific compressor, which I’m not aware of any examples of. (Please tell me if about some if you know them!) A bit academic, but if you check out this paper from 2015 https://arxiv.org/pdf/1502.04623.pdf on a neural network trained on binarised MNIST (28x28 pixel images), Table 2 quotes entropy figures of 116.815 after converting to bits, which would be 28*28/116.815=6.71x compression (if used as a compressor)
Expecting 2 years of research by around 20 of the best DNN researchers on the planet to be compressed into a StackOverflow answer before it has been done seems a fairly large amount to expect. Not a huge fan of the negativity on StackOverflow, but until August 2016 (when the paper this was based on was published - or maybe 2015, with DRAW[2]) people actively working in this area didn't think it was possible. Also, the single answer there certainly didn't say anything like it was a stupid idea. I don't think the author of that answer knew much about autoencoders. Also^2, your question isn't really anything like what this addresses. Your question concentrates on the idea of…
compressing a large set of images, and sharing some kind of representation. That certainly is possible without using a ML approach. And yes, autoencoders have been around for a long time. But hoverboards are a great idea, too. [1] This paper has 7 authors, Gregor et al has 5 authors, DRAW has 6. [2] https://arxiv.org/pdf/1502.04623v2.pdf
Guy, the only thing abundantly clear is that you are full of dung. Your shameless self-promotion may piss-off the police chief here - murbard2, so be more careful. Ten digits better classified, out of 10000? With a structure more complicated than a human DNA vs two lines of code? Congratulations! Or, and how are your "stairways-to-heaven" even remotely universal? Show us something these networks have generated like the VAE or Gibbs/ACE papers? Perhaps you can show some density estimation results as in http://arxiv.org/abs/1502.04623 or http://arxiv.org/abs/1508.06585? Oja and Hyvarinen are great guys and have left their names in the pantheon of neural nets. But it is…
time for you and the other 12 people who live there, to shake off the legacy of ICA and the obsession with orthogonality: Andrew Ng and company showed years ago that it is not needed and is in fact detrimental http://ai.stanford.edu/~quocle/LeKarpenkoNgiamNg.pdf . Read it! Also, too much dung smells bad in the arctic summer, murbard2 here prefers comics and won't be reading your installments of 25+ page spaghetti any time soon. At least Oja and Hyvarinen know how to write.
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.
David Manheim, Scott Garrabrant
There are several distinct failure modes for overoptimization of systems on the basis of metrics.