Skip to content
The methodOpen data
Back to the collection

Neural Machine Translation of Rare Words with Subword Units

Rico Sennrich, Barry Haddow, Alexandra Birch

2015 · arXiv

Computer visionAlgorithms
Read the paper
HN score
0
Mentions
4
Readers
4
Months
4
What’s in a score?

From the abstract

Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character n-gram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English-German and English-Russian by 1.1 and 1.3 BLEU, respectively.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

nl
Neutral mention · rule
> So how would you explain the increase in token usage, considering the fact that conventionally tokenizers are trained to minimize the token usage within a given vocabulary budget? Just modeling whitespace as its own token would seem to explain the increase. > Qwen3.6 tokenizer: "pretokenizer" That's the pre-tokenizer, not the tokenizer. That is mostly a performance optimization that lets the memory requirements for the BPE tokenizer be a lot less. > I could see a world where you could build a better tokenizer training algorithm which would be able to better take the natural morphology of the underlying text into account. The reason everyone went to BPE was because it was so dramatically better than morphology based tokenizers. See the BPE paper: https://arxiv.org/abs/1508.07909 BPE already learns morphology because it sees the raw bytes.
Original comment Claude Token Counter, now with model comparisons
godelski
Neutral mention · rule
I'm not talking about Tokenization. It can get confusing because we usually role tokenization and embedding up as a singular process but the tokenization is the translation of our characters into numeric representations. There's self discovery of what the atomic units should be (bounded by our vocabulary size). The process is, at a high level: string -> integer -> vec<float>. You are learning the string splits, integer IDs, and vector embeddings. You are literally building a dictionary. The BPE paper is a good place to start[0], but it is far from the place we are now. The embeddings is this data in that latent representation space. > All these models are trained for…
Read the full comment
semantic similarity Citation needed... There's no real good measure of semantic similarity so it would be really naive to assume that this must be happening. There is a natural pressure for this to occur because words are generated in a bias way, but that's different than saying they're trained to be semantically similar. There's even a bit of discussion about this in the Word2Vec paper[1], but you should also follow some of the citations to dig deeper. You need to think VERY carefully about the vector basis[2]. You can very easily create an infinite number of basis vectors that are isomorphic to the standard cartesian coordinate. We usually use [[1,0],[0,1]], but there's no reason you can't use some rotation like [[1/sqrt(2), -1/sqrt(2)],[1/sqrt(2),1/sqrt(2)]]. Our (x,y) space is isomorphic to our new (u,v) space but traveling along our u basis vector is not equivalent to traveling along the x basis vector (\hat{i}) or even the y (\hat{j}). You are traveling along them equally! u is still orthogonal to v and x is still orthogonal to y but it is a rotation. we can also do something more complex like using polar coordinates. All this stuff is equivalent! They all provide linearly independent unit vectors that span our field. The point is, the semantics is a happy outcome, not a guaranteed or even specifically trained for outcome. We should expect it to happen frequently because of hour our languages evolved but the "god" "dog" example perfectly illustrates how this is naive. You *CANNOT* train for semantic similarity until you *DEFINE* semantic similarity. That definition needs to be a strong rigorous mathematical one. Not an ad-hoc Justice Potter "know it when I see it" kinda policy. The way they are used in relation to other words is definitely not well aligned to semantics. I can talk about cats and dogs or cats and potatoes all day long. The real similarity we'll come up with there is nouns and that's not much in the way of semantics. Even the examples I gave aren't strictly nouns. Shit gets real fucking messy real fast[3]. It's not just English, it happens in every language[4] We can get WAY more into this, but no, sorry, that's not how this works. [0] https://arxiv.org/abs/1508.07909 [1] https://arxiv.org/abs/1301.3781 [2] https://en.wikipedia.org/wiki/Basis_(linear_algebra) [3] I'll leave you with my favorite example of linguistic ambiguity Read rhymes with lead and lead rhymes with read but read doesn't rhyme with lead and lead doesn't rhyme with read [4] https://en.wikipedia.org/wiki/Lion-Eating_Poet_in_the_Stone_...
Original comment A visual exploration of vector embeddings
loveparade
Neutral mention · rule
They are. Nobody in the AI research community expected that scaling up models several orders of magnitude would make them that good. Even the original selling point of the Transformer paper was "can be much bigger and faster than RNNs, even if not as good" - Of course everyone was wrong and it's all about the data and scale. Btw, tokens are nothing new in modern models. They've been used since 2015/2016 [0] [0] https://arxiv.org/abs/1508.07909
Original comment The Unreasonable Effectiveness of Character-Level Language Models
montebicyclelo
Neutral mention · rule
OpenAI have made their tokenizers public [1]. As someone has pointed out, with BPE you specify the vocab size, not the token size. It's a relatively simple algo, this Huggingface course does a nice job of explaining it [2]. Plus the original paper has a very readable Python example [3]. [1] https://github.com/openai/tiktoken [2] https://huggingface.co/course/chapter6/5?fw=pt [3] https://arxiv.org/abs/1508.07909
Original comment What Are Transformer Models and How Do They Work?

Keep following the thread.

More in Computer vision

More from the same topics.