Tomas Mikolov, Quoc V. Le, Ilya Sutskever
2013 · arXiv
Dictionaries and phrase tables are the basis of modern statistical machine translation systems. This paper develops a method that can automate the process of generating and extending dictionaries and phrase tables. Our method can translate missing word and phrase entries by learning language structures based on large monolingual data and mapping between languages from small bilingual data. It uses distributed representation of words and learns a linear mapping between vector spaces of languages. Despite its simplicity, our method is surprisingly effective: we can achieve almost 90% precision@5 for translation of words between English and Spanish. This method makes little assumption about the languages, so it can be used to extend and refine dictionaries and translation tables for any language pairs.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
3 selected excerpts · 3 total mentions
Models are trained independently for each language. So unfortunately, you cannot directly compare words from different languages using these vectors. If you have a bilingual dictionary, you might try to learn a linear mapping from one language to the other (e.g. see https://arxiv.org/abs/1309.4168 for this approach).
I'm uncertain whether you mean the algorithm or the output. The question is interesting to both however. The most common method for producing word vectors, skip-grams with negative sampling, has been shown to be equivalent to implicitly factorizing a word-context matrix[1]. A related algorithm, GloVe, only uses a word-word co-occurrence matrix to achieve a similar result[2]. You can also view the output as an embedding in a high dimensional space (hence the name word vectors) but more surprisingly you can learn a linear mapping between vector spaces of two languages, which lends it immediately useful in translation. From [3]: "Despite its simplicity, our method is…
surprisingly effective: we can achieve almost 90% precision@5 for translation of words between English and Spanish". [1]: "Neural Word Embedding as Implicit Matrix Factorization" http://papers.nips.cc/paper/5477-neural-word-embedding-as-im... [2]: http://nlp.stanford.edu/projects/glove/ [3]: "Exploiting Similarities among Languages for Machine Translation" - page 2 has an intuitive 2D graphical representation http://arxiv.org/pdf/1309.4168.pdf
Stanford Ph.D. student Richard Socher appreciates the work Google and others are doing to build neural networks that can understand human language. He just thinks his work is more useful ... "We’re actually able to put whole sentences and longer phrases into vector spaces without ignoring the order of the words." Wait, didn't Mikolov et al. (Google) [just figure out][1] how to put entire languages into vector spaces? [1]: http://arxiv.org/abs/1309.4168
More from the same topics.
Martin Kleppmann
The CAP Theorem is a frequently cited impossibility result in distributed systems, especially among NoSQL distributed databases.
Anders Sandberg, Eric Drexler, Toby Ord
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Scott Aaronson
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.