We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
I'm not talking about Tokenization.
It can get confusing because we usually role tokenization and embedding up as a singular process but the tokenization is the translation of our characters into numeric representations. There's self discovery of what the atomic units should be (bounded by our vocabulary size).
The process is, at a high level: string -> integer -> vec<float>. You are learning the string splits, integer IDs, and vector embeddings. You are literally building a dictionary. The BPE paper is a good place to start[0], but it is far from the place we are now.
The embeddings is this data in that latent representation space.
> All these models are trained for…
Read the full comment
semantic similarity Citation needed...
There's no real good measure of semantic similarity so it would be really naive to assume that this must be happening. There is a natural pressure for this to occur because words are generated in a bias way, but that's different than saying they're trained to be semantically similar. There's even a bit of discussion about this in the Word2Vec paper[1], but you should also follow some of the citations to dig deeper.
You need to think VERY carefully about the vector basis[2]. You can very easily create an infinite number of basis vectors that are isomorphic to the standard cartesian coordinate. We usually use [[1,0],[0,1]], but there's no reason you can't use some rotation like [[1/sqrt(2), -1/sqrt(2)],[1/sqrt(2),1/sqrt(2)]]. Our (x,y) space is isomorphic to our new (u,v) space but traveling along our u basis vector is not equivalent to traveling along the x basis vector (\hat{i}) or even the y (\hat{j}). You are traveling along them equally! u is still orthogonal to v and x is still orthogonal to y but it is a rotation. we can also do something more complex like using polar coordinates. All this stuff is equivalent! They all provide linearly independent unit vectors that span our field.
The point is, the semantics is a happy outcome, not a guaranteed or even specifically trained for outcome. We should expect it to happen frequently because of hour our languages evolved but the "god" "dog" example perfectly illustrates how this is naive.
You *CANNOT* train for semantic similarity until you *DEFINE* semantic similarity. That definition needs to be a strong rigorous mathematical one. Not an ad-hoc Justice Potter "know it when I see it" kinda policy. The way they are used in relation to other words is definitely not well aligned to semantics. I can talk about cats and dogs or cats and potatoes all day long. The real similarity we'll come up with there is nouns and that's not much in the way of semantics. Even the examples I gave aren't strictly nouns. Shit gets real fucking messy real fast[3]. It's not just English, it happens in every language[4]
We can get WAY more into this, but no, sorry, that's not how this works.
[0] https://arxiv.org/abs/1508.07909
[1] https://arxiv.org/abs/1301.3781
[2] https://en.wikipedia.org/wiki/Basis_(linear_algebra)
[3] I'll leave you with my favorite example of linguistic ambiguity
Read rhymes with lead and lead rhymes with read but read doesn't rhyme with lead and lead doesn't rhyme with read [4] https://en.wikipedia.org/wiki/Lion-Eating_Poet_in_the_Stone_...
This blog post [0] thinks that it is indeed possible to do arithmetic operations on them. My intuition is that they're vectors after all, and can be added and subtracted like any other vector. A word is just a location in that high-dimensional vector space.
[0] https://blog.dataiku.com/arithmetic-properties-of-word-embed...
EDIT: I guess there are different forms of word embeddings and apparently modern LLMs don't use static word embeddings like word2vec and it's more contextual. Tokens aren't 1:1 with words either of course. I guess it's more complex than "LLMs represent words as vectors". Still though it's a neat trick and is indeed that simple with something like…
Read the full comment
word2vec.
Edit 2: some interesting slides including the bit about semantic analogy and a paraphrase from the original word2vec paper about the king queen thing: https://staff.fnwi.uva.nl/e.kanoulas/wp-content/uploads/Lect...
And the original word2vec paper by Mikolov mentions it https://arxiv.org/abs/1301.3781
The “secret sauce” is Word2Vec. Take a snapshot of all the text you can find on the internet, then with a sliding context window, vectorize each word based on what words are around it. The core assumption is that words with similar context have similar meaning. How you decide which components represent which words in the context is unclear, but it looks like we’re doing some kind of ML training to convince a computer to decide for us. Here’s a paper about the technique which might help: https://arxiv.org/pdf/1301.3781.pdf
Once all the words have vectors you can assume that there’s meaning in there and move on to trying to math these vectors against each other to find…
Read the full comment
interesting correlations. It looks like the scoring for the initial training is based on making the vectors computable in various ways, so you can likely come up with a comparability criteria different than the papers use and get a more useful vectorization for your own purposes. Seems like cosine similarity is good enough for most things though.
The general idea is that you have a particular task & dataset, and you optimize these vectors to maximize that task. So the properties of these vectors - what information is retained and what is left out during the 'compression' - are effectively determined by that task.
In general, the core task for the various "LLM tools" involves prediction of a hidden word, trained on very large quantities of real text - thus also mirroring whatever structure (linguistic, syntactic, semantic, factual, social bias, etc) exists there.
If you want to see how the sausage is made and look at the actual algorithms, then the key two approaches to read up on would probably be Mikolov's…
Read the full comment
word2vec (https://arxiv.org/abs/1301.3781) with the CBOW (Continuous Bag of Words) and Continuous Skip-Gram Model, which are based on relatively simple math optimization, and then on the BERT (https://arxiv.org/abs/1810.04805) structure which does a conceptually similar thing but with a large neural network that can learn more from the same data. For both of them, you can either read the original papers or look up blog posts or videos that explain them, different people have different preferences on how readable academic papers are.
> …we do NOT need new architectures…
You do realise the irony in stating this regarding transformers that arguably made their first appearance as decomposable attention in 2016 [1] and then as transformers in 2017 [2]? It is not as if this is a vanilla RNN straight out of the 90s sweeping the floor with decades of model innovations, rather it looks like we are seeing the rise of a new, simple model category that works remarkably well – akin to how Mikolov et al. reconsidered how to learn vector representations back in 2013 [3] to ingest magnitudes more data than previously possible.
[1]: http://www.aclweb.org/anthology/D/D16/D16-1244.pdf
[2]: http://papers.nips.cc/paper/7181-attention-is-all-you-need
[3]: https://arxiv.org/abs/1301.3781
I would not call myself a model-focused researcher so I would love to play down their importance, but to be intellectually honest one should call out hyperbole where ever one sees it.
Neural networks are all the rage these days. As my knowledge on NNs is quite limited, what I could grasp from the wikipedia article [0] and the original paper [1] is that word2vec uses a vector to represent words in a n-dimensional space.
This seems quite similar to LSA (Latent Semantic Analysis). In LSA, a word is represented by a vector (list / array) of values which represent the number of occurrences of the word in a document. For instance, the list [0, 5, 9, 10] for the word "aeroplane" means than "aeroplane" occurs zero times in D1 (document #1), 5 times in D2, etc. Unlike LSA, word2vec seems to have the vectors values implicit in the neural network weights (?).
The…
Read the full comment
interesting thing on these kind of approaches is that words that share similar contexts, tend to occur in the same documents. In other words, the vectors tend to have approximate values. For instance, it is natural that "aeroplane" tends to occur in the same set of documents as "aeroport".. Tf-Idf [2] does something similar, so these vectors are mostly a data representation of single words.
As for my research, the basis of it was to be able to quantify how "specific" a word or multiword is. The idea is that the more a word/multiword occurs in different contexts (in my case, near different words) the less specific it is. Being able to identify the specificity of words allowed me to know which words/multiwords were probably concepts and which ones weren't. Then, with these concepts I was able to use Tf-Idf to extract document keywords and later I was able to infer broad relationships between concepts (like in if two words tend to co-occur in the same documents and same parts of documents, they are somewhat related).
So, how does word2vec compare to my research?
- I prefer simple statistical approaches like the one I did (which is just counting frequencies and dividing numbers, etc.) than the black-box approach of neural networks. I can tell you why my research provides good results (explained above) but if you read a follow-up paper on word2vec [3] this is what you can find in section 4: "Why does this produce good word representations? Good question. We don’t really know".
- Also, bag-of-box approaches like word2vec and LSA don't tend to represent multiwords (such as "President of the United States of America", which leaves out lots of information on texts.
- Finally, I think there's more possible applications after you know which words/multiwords are informative. With word2vec and LSA you are only representing something similar to frequencies of words in documents, which mostly allows for Information Retrieval-like applications. On the other hand, one of the applications I did with the extracted concepts was to implement a prototype which allowed us to find the definition of a concept on the corpus. The idea was that the definition of a concept was the paragraph where that concept occurred many times but also used other concepts to help the first one to be defined. I don't know how you could do something like that using something like word2vec.
But as I said above, I am not that knowledgeable about word2vec, so I may miss something..
[0] - https://en.wikipedia.org/wiki/Word2vec
[1] - https://arxiv.org/abs/1301.3781
[2] - https://en.wikipedia.org/wiki/Tf%E2%80%93idf
[3] - https://arxiv.org/pdf/1402.3722.pdf
Unpublihed arXiv papers fall in the latter category.
In CS that almost certainly isn't true. I'm most familiar with the NLP field, but there, if you have some kind of embedding of your words/tokens/sentences/something you cite https://arxiv.org/pdf/1301.3781.pdf (Word2Vec, Mikolov).
That paper says there is a follow up paper published at NIPS2013, but I don't think I've ever seen that published.
The field just moves too fast to wait for conferences anymore.
Geoff Hinton refers to thought vectors performing reasoning by analogy using algebra [1] in his Royal Society Lecture.
The other widely reported vector algebras in a semantic space were discovered by Mikolov et al when producing ~300 dimensional vectors for a billion word Wikipedia corpus.
If one performs vector algebra and ~= is near by cosine distance then using Mikolov's Vectors[3].
King - Man + Woman ~= Queen
France - Paris + Gernmany ~= Berlin
Surprisingly this works for other modalities, Chintala, Radford & Metz found a latent semantic space in images, that adds vectors for glasses or smiles to peoples faces. [4] With a generative model new images can be…
Read the full comment
created as outlined in this blog post by Soumith [5]
Karpathy shows trained nets can be assembled like lego across modalities, slice off the classifier to reveal the rich semantic 'thought vector' layer of an Imagenet trained Alexnet, plug in a RNN sentence generator using word2vec and ( some over simplification ... ) you get a convincing image captioner [6].
The thought vectors are akin to high level representations of the world and can cross modalities . Text to Images using thought Vectors ( from hnnews discussion [7] )
So the vectors of though are in some way a an AI mentalese or encoding of a symbolic representation of the world derived from the data and can ( again drastic over simplification ) transfer modalities and even between previously unlinked languages [8]
Also see Anything2Vec https://gab41.lab41.org/anything2vec-e99ec0dc186
[1] https://youtu.be/izrG86jycck?t=25m58s
[2] The paper Geoff Hinton is reffering to : Sequence to Sequence Learning with Neural Networks by Ilya Sutskever, Oriol Vinyals, Quoc V. Le https://arxiv.org/abs/1409.3215
[3] Efficient Estimation of Word Representations in Vector Space by Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean
https://arxiv.org/abs/1301.3781
[4] Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Alec Radford, Luke Metz, Soumith Chintala https://arxiv.org/abs/1511.06434
[5] https://code.facebook.com/posts/1587249151575490/a-path-to-u...
[6] SF Machine Learning: Automated Image Captioning with ConvNets and Recurrent Nets by Karpathy https://youtu.be/ZkY7fAoaNcg?t=38m31s
[7] https://news.ycombinator.com/item?id=12366684
[8] https://github.com/Babylonpartners/fastText_multilingual
As pointed out elsewhere, the results are pretty accurate, not nearly perfect but enough to open up new applications. http://arxiv.org/pdf/1301.3781.pdf
Word2vec is exciting because it has a useable amount of accuracy for a relatively small amount of cleverness on the part of the human. You can just throw a huge corpus at it and it performs reasonably well.
Mikolov, et al. 2013 [1] do a proper evaluation of this. E.g. they found that the skip-ngram model has a 50.0% accuracy for semantic analogy queries and 55.9% accuracy for syntactic queries.
word2vec comes with a data set that you can use to evaluate language models.
[1] http://arxiv.org/pdf/1301.3781.pdf
I'm not associated with the OP, but I have played with word2vec.
Word2vec can be viewed as dimensionality reduction. In some sense, the dimensionality of words is the same as the vocabulary size, if a one-hot encoding is used (which is most common). For word2vec specifically, the output vector dimension is a parameter specified during training, so you can choose it however you want. I suspect the OP chose something fairly large (>256) because of the performance boost.
Nearest neighbor methods work well with word2vec words: https://code.google.com/p/word2vec/#Word_clustering
The paper associated with word2vec is fairly easy to read and understand, even if you don't have a background in NLP or neural networks: http://arxiv.org/pdf/1301.3781.pdf
Very cool. I missed the original word2vec software discussion back in August: https://news.ycombinator.com/item?id=6216044
And the paper itelf is a very worthwhile read: http://arxiv.org/abs/1301.3781
Yeah, the papers linked in the references are probably a better place to start than the readme (though I'm not sure how closely aligned this implementation is with that research, but the paper is still a good read), especially [1]
[1] http://arxiv.org/pdf/1301.3781.pdf
A Hinton inspired speculation about a Robot that plans & communicates.
I posit that general A.I. must be embodied to develop, it must learn from the real world by interacting with it and learn by trial and error.
Deep learning is a huge leap for AI. It learns best from raw data. Babies learn motor control & input interpretation first[1]. Moravec's paradox proposes learning these basics are the hard foundation that higher levels of reason build on.
Abeel and Levine's work[2] where robots learn robust functions to map from pixels to motor torques for domestic tasks show solutions built on features learnt from raw data are more resilient than hand-engineered ones. These…
Read the full comment
trajectories through motor-torque space to produce a desired goal are intriguingly like planning.
Mikolov's[3] Word2vec & Radford's Image Vectors[4] produce semantic spaces where vector math is akin to reasoning. Higher level control could be achieved through vector algebras deriving actions and things directly from the robots internal vector space of motor-torques and raw-pixels
Thus internal vectors are like thoughts[5] which can interpret, reason, plan, and control.
Karpathy shows internal vectors can bridge modalities from words to pictures[6]. If the higher level control vectors of actions and things could translate to the modality of language as verbs and nouns, perhaps the robot could discuss its plans, and receive orders.
[1] - Thelen's Infant Development http://www.ncbi.nlm.nih.gov/pubmed/17107442
[2] - Levine, Finn, Darrel, Abeel http://arxiv.org/abs/1504.00702 &
[2b] - Distributed Grasp Learning http://googleresearch.blogspot.co.uk/2016/03/deep-learning-f...
[3] - Mikolov, Chen, Corrado, Dean http://arxiv.org/abs/1301.3781
[4] - Radford, Metz, Chintala https://github.com/Newmu/dcgan_code#arithmetic-on-faces
[5] - Hinton https://youtu.be/IcOMKXAw5VA?t=32m55s
[6] - Karpathy https://youtu.be/ZkY7fAoaNcg?t=38m31s
The person in the room performing the lookup is a red-herring and can be replaced by suitable algorithm, e.g. a convnet - which could learn the lookup task.
Consciousness resides in the minds that created the lookup tables: they were constructed by conscious beings to map queries to meaningful responses.
The lookup tables are the very sophisticated part of Searle's Chinese Room.
The recent emergent semantic vector algebra discovered in human languages by Mikolov's word2vec [Mikolov et al] demonstrate that some of the computational power of language is inherent in the language ( but only meaningfully interpretable by
a conscious listener.
Meaning requires Consciousness but…
Read the full comment
language is unexpectedly sophisticated and contains a semantic vector space which can answer simple queries ("What is the Capital of...") and analogise ("King is to What as Man is to Woman") algebraicly. [Mikolov et al]
This inherent semantic vector space is discoverable by context encoding large corpii.
Language is a very sophisticated and ancient tool that allows some reasoning to be performed by simple algebraic transformations inherent to the language.
-
[Mikolov et al] :
Efficient Estimation of Word Representations in Vector Space
Mikolov, Chen, Corrado, & Dean
http://arxiv.org/abs/1301.3781
& https://code.google.com/p/word2vec/
Here's a chronological reading list of techniques
Traditional NLP -> 2012, IBM Watson papers[1]
Watson is probably the pinnacle of "traditional" style NLP (ie, Tokenization/Lemmatization/Part of Speech Tagging/Framing/Knowledge Engineering/etc)
Word2Vec -> 2013, Google paper[2]
Word2Vec kind of exploded everyone's mind for a while.
Subgraph Embedding -> 2015, Facebook AI Group [3][4]
[1] http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=617771...
[2] http://arxiv.org/pdf/1301.3781.pdf
[3] https://research.facebook.com/publications/1473550739586509/...
[4] http://arxiv.org/abs/1502.05698
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.