Skip to content
The methodOpen data
Back to the collection

Distributed Representations of Sentences and Documents

Quoc V. Le, Tomas Mikolov

2014 · arXiv

AlgorithmsDistributed systemsMachine learning
Read the paper
HN score
0
Mentions
6
Readers
5
Months
5
What’s in a score?

From the abstract

Many machine learning algorithms require the input to be represented as a fixed-length feature vector. When it comes to texts, one of the most common fixed-length features is bag-of-words. Despite their popularity, bag-of-words features have two major weaknesses: they lose the ordering of the words and they also ignore semantics of the words. For example, "powerful," "strong" and "Paris" are equally distant. In this paper, we propose Paragraph Vector, an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents. Our algorithm represents each document by a dense vector which is trained to predict words in the document. Its construction gives our algorithm the potential to overcome the weaknesses of bag-of-words models. Empirical results show that Paragraph Vectors outperform bag-of-words models as well as other techniques for text representations. Finally, we achieve new state-of-the-art results on several text classification and sentiment analysis tasks.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

6 selected excerpts · 6 total mentions

gojomo
Neutral mention · rule
As another small hint of Mikolov-vs-Le divergence: they're the coauthors of the 'Paragraph Vector' paper (https://arxiv.org/abs/1405.4053) applying a slightly-modified version of word2vec to vectorize longer texts, still in a very shallow way. (This technique often goes by the name 'doc2vec', but others things also sometimes get called that, too.) There are some results in that paper, supposedly from exactly the technique described on an open dataset, that no one has ever been able to reproduce – & you can see the effort has frustrated a lot of people, over the years, in different forums. When asked, Mikolov has said, essentially: "I can't reproduce that either – those tests were run & reported by Le, you'lll have to ask him."
Original comment Word2Vec Received 'Strong Reject' Four Times at ICLR2013
aglionby
Neutral mention · rule
Doc2vec is the name of the gensim implementation of this paper [0]. Briefly, it creates document embeddings (document being anything on the level of phrases, to sentences, to paragraphs and beyond) on the basis of either predicting a word given the context words (distributed memory -- DM), or predicting the context words given a word (distributed bag of words -- DBOW). [0] https://arxiv.org/abs/1405.4053
Original comment Understanding word vectors
versteegen
Neutral mention · rule
Sure, Numenta seemed heavily focused on temporal sequences, but in "On Intelligence" Hawkins describes a more general picture. Prediction should be done not only on the past but on the context. The context includes other variables that also have to be predicted (e.g. predict the next word in a sentence based on a prediction of where this conversation is going) I think that predicting the next word in a sentence does actually get to the point of the issue, which is why the Hutter Prize [0] is defined the way it is. The human brain does: (1) operate sequentially on words (2) keeps an internal state that contains semantic information about what's been heard/seen (3) learns…
Read the full comment
the semantic representation (4) is excellent at predicting the next word in a sentence, and continually does so (5) it's difficult to argue it's not learning to predict the next word in a sentence (ie optimised to that task). Looking at that list, I'd say the only differences between what the brain is doing and what algorithms like these (in general) are doing are just the particular algorithms for learning the semantic representations in working memory and performing prediction, and less importantly the form of the input (humans have more cues available, including taking actions to get more information) (Admittedly parsing a complicated sentence can require jumping backwards, but spoken language will have a very limited nesting level, and this just means including a small piece of the input in the working memory.) In the last week I've actually been reading about algorithms for this (though I didn't know NuPIC was applied to sentences, thanks very much for mentioning it). Aside from the range of RNN-based systems that mietek mentioned, here are a couple more recent papers, related to word2vec, on representing the semantic and syntactic content of a sentence as vectors: [1], [2]. Both are based on optimising for prediction of the next words in a sequence. Now I agree it's dubious to try to reduce a sentence to a small fixed length representation, whether that's e.g. 2400 real numbers in [2], or a pattern of neuron activations in a RNN, but from the experimental results (which admittedly aren't always so great) they seems to be encoding something meaningful, though far too much information gets thrown away. [0] http://prize.hutter1.net [1] Quoc V. Le, Tomas Mikolov, 2014, Distributed Representations of Sentences and Documents, http://arxiv.org/abs/1405.4053 [2] Ryan Kiros et al, 2015, Skip-Thought Vectors, http://arxiv.org/abs/1506.06726 Edit: added a bit more about context
Original comment Why Neurons Have Thousands of Synapses: a Theory of Sequence Memory in Neocortex
gojomo
Neutral mention · rule
The gensim library now offers the 'Paragraph Vector' [1] approach to create vectors for longer ranges of text. It's available in a class named Doc2Vec – but I don't think that's what is being used here. The Paragraph Vector approach can give interesting results for document-similarity, including similarity after certain 'algebraic'-like additions/subtractions of other topics/word-concepts. [2] [1] http://arxiv.org/abs/1405.4053 [2] http://arxiv.org/abs/1507.07998
Original comment Using Doc2Vec to Suggest SubReddits

Keep following the thread.

More in Algorithms

More from the same topics.