In recent years, supervised learning with convolutional networks (CNNs) has seen huge adoption in computer vision applications. Comparatively, unsupervised learning with CNNs has received less attention. In this work we hope to help bridge the gap between the success of CNNs for supervised learning and unsupervised learning. We introduce a class of CNNs called deep convolutional generative adversarial networks (DCGANs), that have certain architectural constraints, and demonstrate that they are a strong candidate for unsupervised learning. Training on various image datasets, we show convincing evidence that our deep convolutional adversarial pair learns a hierarchy of representations from object parts to scenes in both the generator and discriminator. Additionally, we use the learned features for novel tasks - demonstrating their applicability as general image representations.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
I posted a list of papers on twitter, and will be posting a summary for each of them as well. here is the list
https://twitter.com/vrungta/status/1623343807227105280
Unlocking the Secrets of AI: A Journey through the Foundational Papers by @vrungta (2023)
1. "Attention is All You Need" (2017) - https://arxiv.org/abs/1706.03762 (Google Brain)
2. "Generative Adversarial Networks" (2014) - https://arxiv.org/abs/1406.2661 (University of Montreal)
3. "Dynamic Routing Between Capsules" (2017) - https://arxiv.org/abs/1710.09829 (Google Brain)
4. "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks" (2016) -…
Read the full comment
https://arxiv.org/abs/1511.06434 (University of Montreal)
5. "ImageNet Classification with Deep Convolutional Neural Networks" (2012) - https://papers.nips.cc/paper/4824-imagenet-classification-wi... (University of Toronto)
6. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2018) - https://arxiv.org/abs/1810.04805 (Google)
7. "RoBERTa: A Robustly Optimized BERT Pretraining Approach" (2019) - https://arxiv.org/abs/1907.11692 (Facebook AI)
8. "ELMo: Deep contextualized word representations" (2018) - https://arxiv.org/abs/1802.05365 (Allen Institute for Artificial Intelligence)
9. "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context" (2019) - https://arxiv.org/abs/1901.02860 (Google AI Language)
10. "XLNet: Generalized Autoregressive Pretraining for Language Understanding" (2019) - https://arxiv.org/abs/1906.08237 (Google AI Language)
11. T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (2020) - https://arxiv.org/abs/1910.10683 (Google Research)
12. "Language Models are Few-Shot Learners" (2021) - https://arxiv.org/abs/2005.14165 (OpenAI)
Geoff Hinton refers to thought vectors performing reasoning by analogy using algebra [1] in his Royal Society Lecture.
The other widely reported vector algebras in a semantic space were discovered by Mikolov et al when producing ~300 dimensional vectors for a billion word Wikipedia corpus.
If one performs vector algebra and ~= is near by cosine distance then using Mikolov's Vectors[3].
King - Man + Woman ~= Queen
France - Paris + Gernmany ~= Berlin
Surprisingly this works for other modalities, Chintala, Radford & Metz found a latent semantic space in images, that adds vectors for glasses or smiles to peoples faces. [4] With a generative model new images can be…
Read the full comment
created as outlined in this blog post by Soumith [5]
Karpathy shows trained nets can be assembled like lego across modalities, slice off the classifier to reveal the rich semantic 'thought vector' layer of an Imagenet trained Alexnet, plug in a RNN sentence generator using word2vec and ( some over simplification ... ) you get a convincing image captioner [6].
The thought vectors are akin to high level representations of the world and can cross modalities . Text to Images using thought Vectors ( from hnnews discussion [7] )
So the vectors of though are in some way a an AI mentalese or encoding of a symbolic representation of the world derived from the data and can ( again drastic over simplification ) transfer modalities and even between previously unlinked languages [8]
Also see Anything2Vec https://gab41.lab41.org/anything2vec-e99ec0dc186
[1] https://youtu.be/izrG86jycck?t=25m58s
[2] The paper Geoff Hinton is reffering to : Sequence to Sequence Learning with Neural Networks by Ilya Sutskever, Oriol Vinyals, Quoc V. Le https://arxiv.org/abs/1409.3215
[3] Efficient Estimation of Word Representations in Vector Space by Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean
https://arxiv.org/abs/1301.3781
[4] Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Alec Radford, Luke Metz, Soumith Chintala https://arxiv.org/abs/1511.06434
[5] https://code.facebook.com/posts/1587249151575490/a-path-to-u...
[6] SF Machine Learning: Automated Image Captioning with ConvNets and Recurrent Nets by Karpathy https://youtu.be/ZkY7fAoaNcg?t=38m31s
[7] https://news.ycombinator.com/item?id=12366684
[8] https://github.com/Babylonpartners/fastText_multilingual
> But what's the point?
Autoencoders don't really have much practical use now, but that's not the point of them. The point is we really want to figure out how to do unsupervised learning well and autoencoders are one of the few ways of doing it.
We want to do unsupervised learning well because most learning that humans do is unsupervised.
The idea behind autoencoding is that by forcing the network to try to learn efficient ways to compress the data, it could learn important features of the data. The fact that the pictures are so blurry means that this doesn't work very well, but that's why it's a research problem.
Autoencoders don't work well, but some unsupervised techniques that extend on them do, and we get impressive results like https://arxiv.org/pdf/1511.06434.pdf (see page 5) where the network learns to generate natural-looking bedroom images.
I'm guessing recent progress in texture synthesis [1] and content generation [2] using deep convolutional nets made this possible. It would be interesting to see what was actually fed into the generating system. That would be a piece of art truly speaking magnitudes.
[1] http://arxiv.org/abs/1505.07376
[2] http://arxiv.org/abs/1511.06434
I don't speak with any authority on this. With word2vec you can do operations like king - man + woman and the nearest vector will be queen. Similarly capital cities will be in a similar relationship to their countries so if you take Rome - Italy + Spain you will be nearest to Madrid. That permits limited reasoning by analogy and there are some analogy benchmark tasks which those word models perform reasonably well on, effectively answering loads of questions like "Madrid is to Spain as Rome is to ____?".
This model is more complex and seems to be using a series of techniques to "disentangle" the parts of the network so it's better at separating identity, i.e. this is a…
Read the full comment
white triangle, from the parts of the network concerning pose, i.e. it's rotated 20 degrees clockwise. See this paper[0] for definitions. So it can take an input triple and see what the transformation is that's been done to the first two images, and then apply that transformation, disentangled from the identity of the what the transformation was applied to, to a third image. The vector of the third image, with the transformation applied to it, is then visualised by the decoder network. So the key is to not just combine some vectors otherwise you'd end up with a mess, the key is to disentagle the identity so that the relevant part of the vector associated with the change, such as rotation, can be transferred.
I think the link between this kind of operation and word2vec is clearer in the paper Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[1] (submitted for ICLR 2016 it seems[2]) which shows more explicitly the vectors being added and subtracted for images "smiling woman" - "neutral woman" + "neural man" = a series of generated images of a smiling man. Woman is the identity in the first two, by subtracting the identity is effectively removed and the "smile" is largely what remains in the vector to be transferred onto the neutral man identity as a transformation.
I suppose the more general point is that we are going to see more and more complex use of various kinds of thought vectors[3] and more and more attempts to try to improve their accuracy. This area is just going to keep exploding I think because the practical applications, such as they showed in this paper for animations, will keep driving work on it in multiple domains. The current pace of research, which has led to[4], and the results in this paper and in [1] are just mind-boggling.
[0] http://arxiv.org/abs/1210.5474 sections 1+2 have a good overview, and 4 has related work
[1] https://github.com/Newmu/dcgan_code
[2] http://arxiv.org/abs/1511.06434 (first submitted in 19th Nov 2015)
[3] http://arxiv.org/abs/1506.06726
[4] http://www.arxiv-sanity.com/
(1) Adversarial learning is unsupervised and works great.
Most of language modeling is unsupervised (you predict next word, but it's not real supervision because it's self-supervision).
There're many works in computer vision which are unsupervised and still give more or less reasonable performance.
See f.e. http://arxiv.org/pdf/1511.05045v2.pdf for unsupervised learning in action recognition, also http://arxiv.org/pdf/1511.06434v2.pdf and http://www.arxiv-sanity.com/search?q=unsupervised
(2) ImageNet supervision gives you much information to solve other computer vision tasks. So perhaps we don't need to learn everything in unsupervised manner, we might learn most features…
Read the full comment
relevant for most tasks using several supervision tasks. It is kind of cheating but very reasonable one.
Moreover,
(3) We observe now just fantastic decrease of perplexity (btw, it's all unsupervised = self-supervised). It's quite probable that in the very near future neural chat bots write reasonable stories, answer intelligibly with common sense, discuss things. All of this would be just a mere consequence of low enough perplexity. If neural net says smth inconsistent it means that it gives too much probability to some inappropriate words i.e, it's perplexity isn't optimized yet.
(4) It's quite probable that it would open a finish line for human-level AI. AI would be able to learn from textbooks, scientific articles, video lectures. Btw, http://arxiv.org/pdf/1602.03218.pdf gives a potential to synthesize IBM Watson with deep learning. May be, the finish line to human level AI has been opened already.
I've been trying to read a paper a day since midsummer. These are a few of the interesting, for me personally, since then:
Generating Sequences With Recurrent Neural Networks - http://arxiv.org/abs/1308.0850
Older one, but important to understand deeply since other recent ideas have come from this!
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks - http://arxiv.org/abs/1511.06434
Unitary Evolution Recurrent Neural Networks - http://arxiv.org/abs/1511.06464
State of the Art Control of Atari Games Using Shallow Reinforcement Learning - http://arxiv.org/abs/1512.01563
Interesting discussion in section 6.1 on the…
Read the full comment
shortcomings/issues of DQN done by Deepmind
Spectral Representations for Convolutional Neural Networks - http://arxiv.org/abs/1506.03767
Deep Residual Learning for Image Recognition - http://arxiv.org/abs/1512.03385
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) - http://arxiv.org/abs/1511.07289
I wish they did more comparisons between similar network architecture with only the units swapped out. Eg. Alexnet, Relu vs Alexnet, Elu.
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models - http://arxiv.org/abs/1511.09249
Just a few from my list :)
One weird trick for parallelizing ConvNets: http://arxiv.org/abs/1404.5997
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks:
http://arxiv.org/abs/1511.06434
A Neural Network of Artistic Style:
http://arxiv.org/abs/1508.06576
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.