The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
I find the title of the article rather exaggerating...
As of the first difference pointed out in the article, one of the CS224D lectures on word2vec did addressed it:
https://youtu.be/aRqn8t1hLxs?t=2650
It was also mentioned later in the lecture that having two vectors representing each word is meant to make the optimisation easier (so it's kind of a trick); at the end, the two vectors learnt will have to be averaged over in order to reach a single vector for each word.
To be fair, the fact that each word is represented by two vectors was also mentioned in the original paper describing word2vec:
https://arxiv.org/pdf/1310.4546.pdf
On page 3, just beneath equation (2).
Why so surprised?
Links to the relevant papers:
Bag of Tricks for Efficient Text Classification: https://arxiv.org/abs/1607.01759v2
Enriching Word Vectors with Subword Information: https://arxiv.org/abs/1607.04606
Both fantastic papers. For those who aren't aware, Mikolov also helped create word2vec.
One curious thing: this seems to use heirarchal softmax instead of the "negative sampling" described in their earlier paper http://arxiv.org/abs/1310.4546, despite that paper reporting that "negative sampling" is more computationally efficient and of similar quality. Anyone know why that might be?
In my experience it also works fairly well when training e.g. on newspaper text.
It's not really that surprising when you think about how it works. Similar words cluster together in vector space in some dimensions. E.g. 'Paris' and 'Berlin' will both have capital-ish contexts. However, they also are different in some ways, e.g. Paris will have France-ish contexts and Berlin German-ish contexts.
The 2-dimensional PCA projection in figure 2 of one of Mikolov's papers [1] gives an intuition why substraction/addition generally works.
[1] http://arxiv.org/pdf/1310.4546.pdf
Such a fluffy article.
Here is what he's publishing: http://arxiv.org/find/cs/1/au:+Hinton_G/0/1/0/all/0/1
This seems to be a good introduction to the topic: http://arxiv.org/pdf/1310.4546v1.pdf
This is about paragraph vectors: http://arxiv.org/abs/1507.07998 http://arxiv.org/abs/1405.4053
Of course there's not really enough info in this article to really tell what he's talking about.
A Guardian article from May [1] has a bit more about what he is meant by "meaning space":
> Hinton said that the idea that language can be deconstructed with almost mathematical precision is surprising, but true. “If you take the vector for Paris and subtract the vector for France and add Italy, you get Rome,” he said. “It’s quite remarkable.”
... which makes it sounds like this builds off of the "word2vec" [2,3] work that came out in 2013. But there must be something else new (maybe at the sentence level?) to get from there to logic and natural conversation.
[1] http://www.theguardian.com/science/2015/may/21/google-a-step...
[2] https://code.google.com/p/word2vec/
[3] http://arxiv.org/abs/1310.4546
Of course. I thought about using wit.ai a few months ago, but I found a lot of issues in how they approach parsing and classification. Specifically, training it is a very manual process, and customization and tweaking is very hard.
I've built my own API which can do entity and language parsing. It relies on utilizing word embeddings (such as those described here: http://arxiv.org/pdf/1310.4546.pdf).