We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded passage. Sentences that share semantic and syntactic properties are thus mapped to similar vector representations. We next introduce a simple vocabulary expansion method to encode words that were not seen as part of training, allowing us to expand our vocabulary to a million words. After training our model, we extract and evaluate our vectors with linear models on 8 tasks: semantic relatedness, paraphrase detection, image-sentence ranking, question-type classification and 4 benchmark sentiment and subjectivity datasets. The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice. We will make our encoder publicly available.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
I don't speak with any authority on this. With word2vec you can do operations like king - man + woman and the nearest vector will be queen. Similarly capital cities will be in a similar relationship to their countries so if you take Rome - Italy + Spain you will be nearest to Madrid. That permits limited reasoning by analogy and there are some analogy benchmark tasks which those word models perform reasonably well on, effectively answering loads of questions like "Madrid is to Spain as Rome is to ____?".
This model is more complex and seems to be using a series of techniques to "disentangle" the parts of the network so it's better at separating identity, i.e. this is a…
Read the full comment
white triangle, from the parts of the network concerning pose, i.e. it's rotated 20 degrees clockwise. See this paper[0] for definitions. So it can take an input triple and see what the transformation is that's been done to the first two images, and then apply that transformation, disentangled from the identity of the what the transformation was applied to, to a third image. The vector of the third image, with the transformation applied to it, is then visualised by the decoder network. So the key is to not just combine some vectors otherwise you'd end up with a mess, the key is to disentagle the identity so that the relevant part of the vector associated with the change, such as rotation, can be transferred.
I think the link between this kind of operation and word2vec is clearer in the paper Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[1] (submitted for ICLR 2016 it seems[2]) which shows more explicitly the vectors being added and subtracted for images "smiling woman" - "neutral woman" + "neural man" = a series of generated images of a smiling man. Woman is the identity in the first two, by subtracting the identity is effectively removed and the "smile" is largely what remains in the vector to be transferred onto the neutral man identity as a transformation.
I suppose the more general point is that we are going to see more and more complex use of various kinds of thought vectors[3] and more and more attempts to try to improve their accuracy. This area is just going to keep exploding I think because the practical applications, such as they showed in this paper for animations, will keep driving work on it in multiple domains. The current pace of research, which has led to[4], and the results in this paper and in [1] are just mind-boggling.
[0] http://arxiv.org/abs/1210.5474 sections 1+2 have a good overview, and 4 has related work
[1] https://github.com/Newmu/dcgan_code
[2] http://arxiv.org/abs/1511.06434 (first submitted in 19th Nov 2015)
[3] http://arxiv.org/abs/1506.06726
[4] http://www.arxiv-sanity.com/
Sure, Numenta seemed heavily focused on temporal sequences, but in "On Intelligence" Hawkins describes a more general picture. Prediction should be done not only on the past but on the context. The context includes other variables that also have to be predicted (e.g. predict the next word in a sentence based on a prediction of where this conversation is going)
I think that predicting the next word in a sentence does actually get to the point of the issue, which is why the Hutter Prize [0] is defined the way it is.
The human brain does: (1) operate sequentially on words (2) keeps an internal state that contains semantic information about what's been heard/seen (3) learns…
Read the full comment
the semantic representation (4) is excellent at predicting the next word in a sentence, and continually does so (5) it's difficult to argue it's not learning to predict the next word in a sentence (ie optimised to that task).
Looking at that list, I'd say the only differences between what the brain is doing and what algorithms like these (in general) are doing are just the particular algorithms for learning the semantic representations in working memory and performing prediction, and less importantly the form of the input (humans have more cues available, including taking actions to get more information)
(Admittedly parsing a complicated sentence can require jumping backwards, but spoken language will have a very limited nesting level, and this just means including a small piece of the input in the working memory.)
In the last week I've actually been reading about algorithms for this (though I didn't know NuPIC was applied to sentences, thanks very much for mentioning it). Aside from the range of RNN-based systems that mietek mentioned, here are a couple more recent papers, related to word2vec, on representing the semantic and syntactic content of a sentence as vectors: [1], [2]. Both are based on optimising for prediction of the next words in a sequence. Now I agree it's dubious to try to reduce a sentence to a small fixed length representation, whether that's e.g. 2400 real numbers in [2], or a pattern of neuron activations in a RNN, but from the experimental results (which admittedly aren't always so great) they seems to be encoding something meaningful, though far too much information gets thrown away.
[0] http://prize.hutter1.net
[1] Quoc V. Le, Tomas Mikolov, 2014, Distributed Representations of Sentences and Documents, http://arxiv.org/abs/1405.4053
[2] Ryan Kiros et al, 2015, Skip-Thought Vectors, http://arxiv.org/abs/1506.06726
Edit: added a bit more about context
I have shivers going down my spine at the idea of feeding, say, skip-thought vectors representing sentences[1] into stack-augmented recurrent neural nets as proposed in Facebook's paper[2], for all sorts of interesting tasks.
--
[1] http://arxiv.org/pdf/1506.06726v1.pdf
[2] http://arxiv.org/pdf/1503.01007.pdf
This is not about word2vec, an algorithm for learning vector representations of words.
This is about learning vector representations of thoughts (ideas, concepts, etc.) in an unsupervised manner.
For reference, two months ago some of Hinton's colleagues at the University of Toronto published a paper describing an unsupervised algorithm for learning vector representations of sentences: http://arxiv.org/pdf/1506.06726v1.pdf
Once trained, this algorithm is given a sentence and asked to find similar sentences in a corpus of half a million sentences, and it produces impressive results. For example, when asked to find the sentence with most similar vector representation of…
Read the full comment
sentence 1, it finds sentence 2:
* Sentence 1 (sentence given to the trained algorithm): "He ran his hand inside his coat, double-checking that the unopened letter was still there."
* Sentence 2 (sentence with most similar vector representation found by algorithm in corpus): "He slipped his hand between his coat and his shirt, where the folded copies lay in a brown envelope."
The paper has more examples. The vector representations learned by this type of algorithm truly appear to be capturing the meaning of sentences.
This looks promising.
--
Edit: Code used in the paper I mention is/will be here: https://github.com/ryankiros/skip-thoughts
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.