Skip to content
The methodOpen data
Back to the collection

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Yonghui Wu, Mike Schuster, Zhifeng Chen

Show all 31 authors

Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, Jeffrey Dean

2016 · arXiv

Other research
Read the paper
HN score
0
Mentions
7
Readers
7
Months
6
What’s in a score?

From the abstract

Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

7 selected excerpts · 7 total mentions

tommit
Neutral mention · rule
You definitely have good points, but since the post I responded to gave Alexa as an initial example, I kind of took that as baseline. I can't really argue with anecdotes, but I can ensure you that a whole lot has changed in Machine Translation in the past 7 years, starting with Google introducing Neural Machine Translation[1] to replace their statistical model, which would often behave in the way you described. That's why I specifically included idioms, which hadn't really been possible until then. It's not yet perfect, but it's crazy good at what it does and only getting better. I know AI Dungeon was really wonky, but I also said it gives us a glimpse of what may be…
Read the full comment
possible. Because the way it interprets natural language is really something else (granted, that's the underlying model, of course). It's really a product still entirely in its infancy. And I don't think AI Dungeon will take the world by storm much more than it has done thus far, but I could imagine countless applications for a similar but improved technology. I don't know, OP was asking for products and services, and I gave some examples. Are they flawless? No. Is most technology flawless? No. Will there be growing market for somewhat imperfect AI-based applications? I surely think so. In the end, humans aren't flawless either. [1] https://en.wikipedia.org/wiki/Google_Neural_Machine_Translat..., https://arxiv.org/pdf/1609.08144.pdf
Original comment The new business of AI and how it’s different from traditional software
anewhnaccount2
Neutral mention · rule
Compare this to one of Google's blog post promoting their MT research: https://research.googleblog.com/2016/09/a-neural-network-for... It is: 1) More accurate, compared to hyperbole like e.g. "Bridging the Gap between Human and Machine Translation" we have right there in the title the domain: news. 2) A more impressive result. This result is on an independently set up evaluation framework, compared to Google's which used their own framework. Compare further the papers: https://arxiv.org/pdf/1609.08144.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0... These researcher appear to have been much clearer about what they're actually claiming, and also used more standard evaluation tools (Appraise) and methodology rather than something haphazardly hacked together.
Original comment Using AI to match human performance in translating news from Chinese to English
ma2rten
Neutral mention · rule
No offense, but this person has no idea what they are talking about. Google published a paper that explains exactly what they are doing and provides metrics as well [1,2]. It's true that neural networks can not be easily applied to natural languages, but there are less obvious ways of applying them (namely Embeddings, LSTMs, Attention) and provided you have enough data and computational resources they give so much better results than any other method that it doesn't even help anymore to combine them. [1] https://arxiv.org/abs/1609.08144 [2] https://arxiv.org/abs/1611.04558 EDIT: I also don't want to give the impression that Deep Learning can solve every NLP problem. We…
Read the full comment
are still far away from passing the Turing test. It's true as well in my opinion that Google's Machine Translation is oversold. It's best at what it's trained and evaluated for: translating individual sentences from news sources. EDIT 2: There are some tasks where traditional methods can work better, e.g. text classification on long documents. That's mainly because deep learning methods are too expensive computationally.
Original comment Why Deep Learning Cannot Be Applied to Natural Languages Easily
dennybritz
Neutral mention · rule
A lot of recent research uses RL to "fine-tune" NLP models. A practical example would be Google's recently announced Machine Translation System (https://arxiv.org/abs/1609.08144). It uses RL to directly optimize BLEU scores on translated sentences. You'll find similar applications in state-of-the art models for chatbots for example. Though I agree, "widely used" may be somewhat of an overstatement. But it's becoming more common. On a side note, I actually think RL makes a lot of sense for many NLP problems and it would be super interesting to build a pure RL approach to language modeling or translation. Nobody has managed to do that quite yet.
Original comment Learning Reinforcement Learning, with Code, Exercises, and Solutions
aub3bhat
Neutral mention · rule
Honestly this comment reads like an emotional outburst, by someone who has zero clue about the domain. Its very well known fact in AI/ML community, that Apple has almost little or no talent nor do they have any major efforts at organizational level (E.g. FAIR at FB, MSR, Brain & Deep Mind at Google). >> But in Machine Translation, Apple is ahead. LOL where did you get this from? I am pretty sure that Google NTM which was put in production yesterday is the state of the art. Also there are metrics e.g. in the report released yesterday you can find BLEU scores on WMT 2014 tasks, and the clear conclusion is that Google is way ahead. Also when it comes to Imagenet or Coco…
Read the full comment
challenges, I don't think Apple has competed in any let alone placing anywhere at top, while FB, MSR and Google all have had top models. http://arxiv.org/pdf/1609.08144v1.pdf >> but the popular press is not discerning enough to make heads or tales out of what they report on, and they end up mis-educating the public. Sorry you are wrong, the sad reality (for Apple), is that they are truly 3-4 years behind FB, MSFT & Google.
Original comment Facebook, Amazon, Google, IBM and Microsoft Create Partnership on AI

Keep following the thread.

More in Other research

More from the same topics.