Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
Thanks for the great comment! For a direct link to the visual saliency maps from the paper (Figure 6) which @nicklo mentions:
http://i.imgur.com/DRfaNxB.png
Being able to see where a neural attention mechanism is fascinating and allows for far more than just introspection. Indeed, one of the earliest and coolest examples are neural attention mechanisms learning to align words between languages without any help[1]!
http://i.imgur.com/J5zFZzN.png
New papers on neural attention seem to be coming out every few weeks. I devoted much of my weekend to a naive implementation of hierarchical attentive memory[2] that promises O(log n) lookup (important for improving the speed of…
Read the full comment
attention based algorithms on large input) and can be "taught" to sort in O(n log n) - really exciting stuff!
For those interested, I highly recommend reading NVIDIA's "Introduction to Neural Machine Translation with GPUs"[3]. The three part overview is a great introduction.
I'd be really happy to see 2016 as the year of neural attention mechanisms :)
[1]: http://arxiv.org/abs/1409.0473
[2]: http://arxiv.org/abs/1602.03218
[3]: https://devblogs.nvidia.com/parallelforall/introduction-neur...
(full disclosure: one of the authors of the paper)
No need stupid moving texts.
CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/
The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/
Understanding LSTM Networks - https://colah.github.io/posts/2015-08-Understanding-LSTMs/
ImageNet Classification with Deep Convolutional Neural Networks - https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436...
Deep Residual Learning for Image Recognition - https://arxiv.org/abs/1512.03385
Multi-Scale Context Aggregation by Dilated Convolutions - https://arxiv.org/abs/1511.07122
Identity Mappings in Deep Residual Networks -…
Read the full comment
https://arxiv.org/abs/1603.05027
Recurrent Neural Network Regularization - https://arxiv.org/abs/1409.2329
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin - https://arxiv.org/abs/1512.02595
Order Matters: Sequence to Sequence for Sets - https://arxiv.org/abs/1511.06391
Neural Machine Translation by Jointly Learning to Align and Translate - https://arxiv.org/abs/1409.0473
Pointer Networks - https://arxiv.org/abs/1506.03134
Attention Is All You Need - https://arxiv.org/abs/1706.03762
The Annotated Transformer - https://nlp.seas.harvard.edu/annotated-transformer/
Neural Turing Machines - https://arxiv.org/abs/1410.5401
A Simple Neural Network Module for Relational Reasoning - https://arxiv.org/abs/1706.01427
Relational Recurrent Neural Networks - https://arxiv.org/abs/1806.01822
Neural Message Passing for Quantum Chemistry - https://arxiv.org/abs/1704.01212
Scaling Laws for Neural Language Models - https://arxiv.org/abs/2001.08361
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism - https://arxiv.org/abs/1811.06965
Keeping Neural Networks Simple by Minimizing the Description Length of the Weights - https://www.cs.toronto.edu/~hinton/absps/colt93.pdf
A Tutorial Introduction to the Minimum Description Length Principle - https://arxiv.org/abs/math/0406077
The First Law of Complexodynamics - https://scottaaronson.blog/?p=762
Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton - https://arxiv.org/abs/1405.6903
Kolmogorov Complexity - https://onlinelibrary.wiley.com/doi/book/10.1002/047174882X
Variational Lossy Autoencoder - https://arxiv.org/abs/1611.02731
Machine Super Intelligence - https://www.vetta.org/documents/Machine_Super_Intelligence.p...
You might be thinking of this article by Sebastian Ruder: https://www.ruder.io/nlp-imagenet/
Note that the author has a background spanning a lot of the timespans/topics discussed - much work in multilingual NLP, translation, and more recently at DeepMind, Cohere, and Meta (in other words, someone with a great perspective on everything in the top article).
Re: Machine Translation, note that Transformers were introduced for this task, and built on one of the earlier notions of attention in sequence models: https://arxiv.org/abs/1409.0473 (2014, 38k citations)
That's not to say there weren't holdouts or people who really were "hurt" by a huge jump in MT capability - just that this is a logical progression in language understanding methods as seen by some folks (though who could have predicted the popularity of chat interfaces).
People did something similar to what you are describing 10 years ago: https://arxiv.org/abs/1409.0473
But it's trained on translations, rather than the whole Internet.
I am not sure if this was the very first paper talking about attention, but https://arxiv.org/abs/1409.0473 from 2014 is a famous one.
What is still missing there, is the idea of self-attention.
I struggled to get an intuition for this, but on another HN thread earlier this year saw the recommendation for Sebastian Raschka's series. Starting with this video: https://www.youtube.com/watch?v=mDZil99CtSU and maybe the next three or four. It was really helpful to get a sense of the original 2014 concept of attention which is easier to understand but less powerful (https://arxiv.org/abs/1409.0473), and then how it gets powerful with the more modern notion of attention. So if you have a reasonable intuition for "regular" ANNs I think this is a great place to start.
The audience of this paper are other researchers who already know the concept of attention, which was very well known already in the field. In such research papers, such things are never explained again, as all the researchers already know this or can read other sources, which are cited, but focus on the actual research questions. In this case, the research question was simply: Can we get away by just using attention and not using the LSTM anymore? Before that, everyone was using both together.
I think learning it following it more this historical development can be helpful. E.g. in this case here, learn the concept of attention, specifically cross attention first. And…
Read the full comment
that is this paper: Bahdanau, Cho, Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate", 2014, https://arxiv.org/abs/1409.0473
That paper introduces it. But even that is maybe quite dense, and to really grasp it, it helps to reimplement those things.
It's always dense, because those papers already have space constraints given by the conferences, max 9 pages or so. To get a better detailed overview, you can study the authors code, or other resources. There is a lot now about those topics, whole books, etc.
Intelligent trial and error.
1. Transformers are an extension of the attention mechanism, a well known LSTM extension that worked well for machine translation (2014, https://arxiv.org/abs/1409.0473). A transformer model is essentially building a multi headed attention module, analogue to CNNs, then stacking several layers of them, analogue to CNNs / stacked LSTMs. (1998, http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf)
2. Transfomers use residual blocks, which were introduced by the ResNet CNN arhitecture (2015, https://arxiv.org/abs/1512.03385). At the time, ResNet was topping the ImageNet benchmarks. This technique helps preventing the vanishing gradient problem…
Read the full comment
during training.
3. Transformers use normalization extensively. Layer normalization and attention normalization. This helps keeping internal vectors in the neighborhood of 1 and prevents vanishing gradient training collapse.
4. Correct initialization of the network vectors also helps preventing the vanishing gradients problem.
5. Unsupervised pretraining was one of the first tricks to make deep networks work (2009, https://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf)
6. Pretraining was used extensively in the vision community for transfer learning, i.e. reusing the weights of a network trained on ImageNet and replace the top layers / loss function to tackle a different problem.
7. Finally, language modelling, that is predicting the next word in a sentence, was a well known technique to make machine translation work better. Researchers were looking for better language modelling techniques using large corpora (2013, https://arxiv.org/abs/1312.3005, https://www.kaggle.com/c/billion-word-imputation)
Seems like the article is mistaken on one part.
Attention was first coined in this paper (as far as I know):
https://arxiv.org/pdf/1409.0473v7.pdf
The second page of the introduction of the "Hierarchical Attention Networks for Document Classification" paper mentioned in the article even cites it.
Combining word embeddings to get a single representation of a sentence in general is a pretty hard problem. Point-wise averaging, addition or summation [0] works alright, but it's not great. Some more recent work looks a using RNNs to combine them; most of the literature I've seen is pretty application-specific, but try [1][2] for flavour.
[0] http://www.aclweb.org/anthology/P/P08/P08-1028
[1] http://arxiv.org/abs/1409.3215
[2] http://arxiv.org/abs/1409.0473
The literature is mostly image based, but as other users have said voice, text, and video have also seen substantial gains. It has also been used to great effect for taxi destination prediction (heterogenous information) [1], drug discovery [2], and general operations on graphs [3].
There are also lots of examples beyond classification - structured prediction such as image segmentation [4], predicting depth, surface normals, and pixels from only RGB [5], text to text translation [6], image captioning [7], Turing machines [8], speech synthesis [9], question answering (Q/A) [10][11], handwriting generation [12], playing Go [13], playing Atari [14], speech recognition [15],…
Read the full comment
drawing things [16] and many more.
[1] http://blog.kaggle.com/2015/07/27/taxi-trajectory-winners-in...
[2] http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it...
[3] http://arxiv.org/abs/1403.6652
[4] http://arxiv.org/abs/1502.03240
[5] http://arxiv.org/abs/1411.4734
[6] http://arxiv.org/abs/1409.0473
[7] http://arxiv.org/abs/1502.03044
[8] http://arxiv.org/abs/1410.5401
[9] http://arxiv.org/abs/1506.02216
[10] http://arxiv.org/abs/1507.04808
[11] http://arxiv.org/abs/1503.08895
[12] http://arxiv.org/abs/1308.0850
[13] http://arxiv.org/abs/1412.3409
[14] https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
[15] http://www.sciencedirect.com/science/article/pii/S0893608014...
[16] https://www.youtube.com/watch?v=Zt-7MI9eKEo
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.