Skip to content
The methodOpen data
Back to the collection

Sequence to Sequence Learning with Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le

2014 · arXiv

Machine learningAlgorithms
Read the paper
HN score
0
Mentions
9
Readers
8
Months
9
What’s in a score?

From the abstract

Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT'14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous best result on this task. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

9 selected excerpts · 9 total mentions

bob1029
Neutral mention · rule
> Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. In context of genetic programming and other non-traditional ML techniques, I've been having difficulty attempting to locate a simple fitness function that reliably proxies natural language string similarity due to this effect. For example, say you use something like common prefix length to measure how close a candidate's output string is to an objective string given an input string. The underlying learner will inevitably start doing things like repeating the input verbatim, especially if the input/output training tuples often share a lot of prefixes. So, you…
Read the full comment
might try doing something like reversing the input to force learning to take a less crappy path [0]. The learner may respond degenerately by inventing a string reversing technique and repeating its prior behavior. So, you iterate again and try something like base64 encoding the input. This might take, but eventually you wind up with so many weird hacks that the learner can't make progress and the meaning of the quantities evaporates. Every metric I've ever looked at gets cheated in some way. The holy grail is probably normalized information distance (approximated by normalized compression distance), but then you have a whole new problem of finding an ideal universal compressor which definitely doesn't exist. [0]: https://arxiv.org/abs/1409.3215 (Figure 1)
Original comment The Leaderboard Illusion
smoldesu
Neutral mention · rule
OpenAI was most certainly not the first organization to develop (or distribute) an LLM model: https://github.com/google-research/bert https://arxiv.org/abs/1810.04805 Failing that, Seq2seq predates OpenAI's incorporation: https://google.github.io/seq2seq/ https://github.com/google/seq2seq https://arxiv.org/abs/1409.3215
Original comment Ask HN: Why is OpenAI still called OpenAI?
danieldk
Neutral mention · rule
This is basically how RNN encoder/decoder architectures worked. The encoder encoded the input as a single vector and the decoder would decode this into a sentence (eg. for machine translation) [1]. However, fixed-length vectors generally required too much 'compression' to represent variable-length sentences, so people started adding attention mechanisms so that the decoder could also attend to the input text. And the seminal Transformer paper by Vaswani and others showed that you only need the attention mechanism and you could ditch the RNN (hence the title 'Attention is all you need') and here we are. So, this has been possible already for quite a long time. [1] https://arxiv.org/pdf/1409.3215.pdf
Original comment What Are Embeddings?
aglionby
Neutral mention · rule
Combining word embeddings to get a single representation of a sentence in general is a pretty hard problem. Point-wise averaging, addition or summation [0] works alright, but it's not great. Some more recent work looks a using RNNs to combine them; most of the literature I've seen is pretty application-specific, but try [1][2] for flavour. [0] http://www.aclweb.org/anthology/P/P08/P08-1028 [1] http://arxiv.org/abs/1409.3215 [2] http://arxiv.org/abs/1409.0473
Original comment Understanding word vectors
deepnet
Neutral mention · rule
Geoff Hinton refers to thought vectors performing reasoning by analogy using algebra [1] in his Royal Society Lecture. The other widely reported vector algebras in a semantic space were discovered by Mikolov et al when producing ~300 dimensional vectors for a billion word Wikipedia corpus. If one performs vector algebra and ~= is near by cosine distance then using Mikolov's Vectors[3]. King - Man + Woman ~= Queen France - Paris + Gernmany ~= Berlin Surprisingly this works for other modalities, Chintala, Radford & Metz found a latent semantic space in images, that adds vectors for glasses or smiles to peoples faces. [4] With a generative model new images can be…
Read the full comment
created as outlined in this blog post by Soumith [5] Karpathy shows trained nets can be assembled like lego across modalities, slice off the classifier to reveal the rich semantic 'thought vector' layer of an Imagenet trained Alexnet, plug in a RNN sentence generator using word2vec and ( some over simplification ... ) you get a convincing image captioner [6]. The thought vectors are akin to high level representations of the world and can cross modalities . Text to Images using thought Vectors ( from hnnews discussion [7] ) So the vectors of though are in some way a an AI mentalese or encoding of a symbolic representation of the world derived from the data and can ( again drastic over simplification ) transfer modalities and even between previously unlinked languages [8] Also see Anything2Vec https://gab41.lab41.org/anything2vec-e99ec0dc186 [1] https://youtu.be/izrG86jycck?t=25m58s [2] The paper Geoff Hinton is reffering to : Sequence to Sequence Learning with Neural Networks by Ilya Sutskever, Oriol Vinyals, Quoc V. Le https://arxiv.org/abs/1409.3215 [3] Efficient Estimation of Word Representations in Vector Space by Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean https://arxiv.org/abs/1301.3781 [4] Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks Alec Radford, Luke Metz, Soumith Chintala https://arxiv.org/abs/1511.06434 [5] https://code.facebook.com/posts/1587249151575490/a-path-to-u... [6] SF Machine Learning: Automated Image Captioning with ConvNets and Recurrent Nets by Karpathy https://youtu.be/ZkY7fAoaNcg?t=38m31s [7] https://news.ycombinator.com/item?id=12366684 [8] https://github.com/Babylonpartners/fastText_multilingual
Original comment When not to use deep learning
nl
Neutral mention · rule
no it wouldn't and it doesn't. Either you would have to strip it down too much to fit on a phone, so it would lose performance (and it's far from state-of-the-art anyway, being an example) or it would not fit on your phone, unless you removed everything else from it (and even then). Yes, it is an example program. Nevertheless, to quote: "Even if you want to transform a sequence to a tree, for example to generate a parsing tree, the same model as above can give state-of-the-art results, as demonstrated in Vinyals & Kaiser et al., 2014" Language modelling isn't the same as translation, but it is closely related. Seq2seq as an approach does achieve close to state-of-the-art…
Read the full comment
for translation[1]. Why do you think it won't fit on your phone? A related seq2seq model in TensorFlow comes in at less than 500kb[2]. The (much more complex) Image Recognition demo is designed to run on Android[3]. The new TensorFlow quantization approach[4] will reduce that size and increase performance even more. [1] http://arxiv.org/pdf/1409.3215.pdf [2] https://github.com/cmusphinx/g2p-seq2seq [3] https://github.com/tensorflow/tensorflow/tree/master/tensorf... [4] https://petewarden.com/2016/05/03/how-to-quantize-neural-net...
Original comment My path to OpenAI
ogrisel
Neutral mention · rule
New neural machine translation architectures are experimenting with pairs of neural encoders / decoders, one pair for each language and a shared language independent vector space for the meaning of all words: http://arxiv.org/abs/1406.1078 http://arxiv.org/abs/1409.3215 http://arxiv.org/abs/1410.8206 So the total number of models is still linear with the number of languages. I do not know whether this new generation of translation models is leveraged by the google translation app though. Also pairs of languages for which their are big amount of parallel training data will still be favored.
Original comment New, more powerful Google Translate app
deepnet
Neutral mention · rule
This work indicates that language translation is vector addition on a lower dimensional semantic manifold that neural networks converge on, derived from the high dimensional whole vocabulary context vectors of the input. It is extended by : Sequence to Sequence Learning with Neural Networks by Sutskever, Vinyals & Le http://arxiv.org/abs/1409.3215 which adds LTSM, a trainable deep neural net type of long term memory - which allows whole sentences to be digested and translated. " Our main result is that on an English to French translation task from the WMT'14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, ...…
Read the full comment
Additionally, the LSTM did not have difficulty on long sentences."[1] another interesting result is the translation of pixels to words that describe the image. Deep Fragment Embeddings for Bidirectional Image Sentence Mapping by Karpathy, Joulin, & Li http://papers.nips.cc/paper/5281-deep-fragment-embeddings-fo... code & a demo is available : http://cs.stanford.edu/people/karpathy/deepimagesent/
Original comment Exploiting Similarities Among Languages for Machine Translation (2013)

Keep following the thread.

More in Machine learning

More from the same topics.