xVal: A Continuous Numerical Tokenization for Scientific Language Models

> Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs - https://arxiv.org/abs/2402.14903 Very interesting paper. It does make sense to me the R2L chunking would be better than L2R chunking. It doesn't actually study single digit tokenization. I am mostly interested in a direct comparison between an LLM wide tokenization vs single digit tokenization. It would be nice to see a direct comparison between similarly trained models. Otherwise it is very hard to get a definitive answer by comparing models with varying sizes, training time, and general strength. > xVal: A Continuous Number Encoding for Large Language Models - https://arxiv.org/abs/2310.02989 I have seen this paper before, but hadn't payed attention to the p10 vs p100 analysis. Its not clear that the findings would be relevant to an LLM like gtp4 though.

Tokenizers aren't considered the "sexy" part of LLMs, but where others see boring, I see opportunity. Papers like xVal[1], point toward specialization strategies in tokenization. Spelling and letter tasks are another problem that could benefit from innovation on the tokenization. LLMs are notoriously bad at counting letters in words or performing simply oulipos of letter omission. GPT-4o, for example, writes a small python program and executes it in order to count letter instances. We all know that tokenization effectively erases knowledge about letters in prompts and directly negatively impacts performance at these tasks, yet we haven't found a way to solve it. 1. https://ar5iv.labs.arxiv.org/html/2310.02989

> there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers. fundamental inability ? No. Current Sota LLM (4o, claude, gemini) woes with arithmetic is not a transformer weakness never mind a large language modelling one. Those benchmarks show that those particular models have problems with accuracy on that many digits, not that LLMs fundamentally cannot be accurate on that many digits. https://arxiv.org/abs/2405.17399 https://arxiv.org/abs/2307.03381 https://arxiv.org/abs/2310.02989 Ultimately, numbers are represented in GPT like systems in a pretty weird way. That way affects how well they can learn to do things like arithmetic and counting and the like but that way isn't a necessary way. You don't have to represent numbers like that.

>But in one sentence, try asking ChatGPT to reverse a string of 20+ digits, like 473936482738338373926. LLMs don't see numbers or letters but tokens. Letter, digit level manipulation is intuitively a hard task. Unsurprisingly, removing that handicap resulted in much better arithmetic abilities. https://arxiv.org/abs/2310.02989 It's pretty telling in my opinion that this is the kind of "proof" people bank hard on. For something that so obviously "does not understand", you'd think providing a task that it would fail that a chunk of humans also wouldn't would be easy.

My apologies if this has already been submitted. I kept the title used by u/yahma on r/LocalLLaMa as it's a good straight to the point title. The paper is here https://arxiv.org/abs/2310.02989

>It has seemed to me that the GPT would be considerably better at numbers if it just considered each digit as a token. Has anyone actually done an experiment to test this? Yeah Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs - https://arxiv.org/abs/2402.14903 xVal: A Continuous Number Encoding for Large Language Models - https://arxiv.org/abs/2310.02989 I believe there's another paper that demonstrates something like also for the likes of spelling, counting etc but i can't remember it.

> You can't really train them to do math unless they've seen previous calculations with similar numbers in their training sets. Generally true before xVal[1], but not exactly true after. Expect fairly basic advancement in the future to solve this problem completely. 1. https://arxiv.org/pdf/2310.02989.pdf

The point is that this is just blatantly wrong. >Asking a token prediction model to do math is like asking a human to do math without doing the math. What's 9 times 9? I can tell you it's 81 from sheer memorization. I can probably invoke 9 x 9 + 1 = 82 without needing to do any calculation either. But if you ask me 32 * 64, that's very difficult to do without doing calculations. Implicitly doing math is not sensible. It's not about memorization. It's trivial to test on instances that would not appear in training and see GPT-4 be better than a human who would attempt the problem without a tool or pad. The biggest problem with LLM arithmetic is tokenization. https://arxiv.org/abs/2310.02989 Other than that, the algorithms it uses for arithmetic will continue to converge during training until it is correct.

GPT-4 can do arithmetic just fine. In fact, despite being severely hobbled by Tokenization (https://arxiv.org/abs/2310.02989), it can perform arithmetic better than you could without a dedicated tool or pad.

It's better (GPT-4) than you could ever manage without an external tool. and that's after being severely hampered by tokenization. https://arxiv.org/abs/2310.02989