Hallucination is Inevitable: An Innate Limitation of Large Language Models

> it can work through the complex, vague machinery of reason with more scaffolding No, it can hold more floating point numbers. I'm not an expert in the field, but I've yet to see a solid rebuttal to this paper; https://arxiv.org/abs/2401.11817

As a general architecture, an LLM also has limitations that can't be improved unless we switch to another, fundamentally different AI design that's non LLM based. There are also limitations due to maths and/or physics that aren't fixable under any design. Outside science fiction, there is no technology whose limitations are all fixable. Here's one: https://arxiv.org/abs/2401.11817?utm_source=chatgpt.com

LLMs "hallucinate" because they are stochastic processes predicting the next word without any guarantees at being correct or truthful. It's literally an unavoidable fact unless we change the modelling approach. Which very few people are bothering to attempt right now. Training data quality does matter but even with "perfect" data and a prompt in the training data it can still happen. LLMs don't actually know anything and they also don't know what they don't know. https://arxiv.org/abs/2401.11817

> True, but no more true than it is if you replace the antecedent with "people". Incorrect. People are capable of learning by observation, introspection, and reasoning. LLMs can only be trained by rote example. Hallucinations are, in fact, an unavoidable property of the technology - something which is not true for people. [0] [0] https://arxiv.org/abs/2401.11817

Why is every discussion about AI like this? We get study after study and example after example showing that AI is unreliable [1], hurts productivity [2], and causes cognitive decline [3]. Yet, time and time again, a group of highly motivated individuals show up and dismiss these findings, not with counter-evidence, but through sheer sophistry. Heck, this is a thread meant to discuss evidence about LLM reliance contributing to cognitive decline. Instead, the conversation quickly derailed into an absurd debate about whether AI coding tools resemble compilers. It's exhausting. [1]: https://arxiv.org/abs/2401.11817 [2]: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o... [3]: https://publichealthpolicyjournal.com/mit-study-finds-artifi...

> Is the layoff-based business model really the best use case for AI systems? No, but it plays well in quarterly earnings. So expect a good bit, I'd say. > The flaws are baked into the training data. Not in terms of attack area. In terms of hallucinations... ish. It's more baked into the structure than the training data. There's increasing amounts of work to ensure well-curated training data. Still, the whole idea of probabilistic selection of tokens more or less guarantees it'll go wrong from time to time. (See e.g. https://arxiv.org/abs/2401.11817) In terms of safety, there's some progress. CaMeL is rather interesting: https://arxiv.org/abs/2503.18813 > "Trust but verify" applies. FWIW, that applies to humans too. Nobody is error-free. And so the question becomes mostly "does the value overshadow the cost of the error", as always.

Post title: Automated reasoning to remove LLM hallucinations --- and yet, the paper that went around in March: Paper Link: https://arxiv.org/pdf/2401.11817 Paper Title; Hallucination is Inevitable: An Innate Limitation of Large Language Models --- Instead of trying to trick a bunch of people into thinking we can somehow ignore the flaws of post-LLM "AI" by also using the still flawed pre-LLM "AI", why don't we cut the salesman BS and just tell people not to use "AI" for the range of tasks it's not suited for.

Much like you can search the internet until you find a source that agrees with you, you can select a set of papers that "confirm" a particular viewpoint, especially in developing fields of research. In this case, the selected papers all support the view LLMs "know what they know" on some internal level, which iiuc is not (yet?) a consensus viewpoint (from my outsider perspective). But from the list alone, you might get that impression. Related: Still no lie detector for language models: probing empirical and conceptual roadblocks - https://link.springer.com/article/10.1007/s11098-023-02094-3 Hallucination is Inevitable: An Innate Limitation of Large Language Models - https://arxiv.org/abs/2401.11817 LLMs Will Always Hallucinate, and We Need to Live With This - https://arxiv.org/abs/2409.05746 (disclaimer, I also have not read any of these papers beyond the title!)

On what basis you think you will avoid hallucination(s) ? LLM will always hallucinate. People did already some research, among which : https://arxiv.org/abs/2409.05746 and https://arxiv.org/abs/2401.11817 for example.

Some things like: https://arxiv.org/abs/2310.18168 https://arxiv.org/abs/2310.06824 There are various reasons an LLM might have incorrect "beliefs" - the input text was false, training doesn't try to preserve true beliefs, quantization certainly doesn't. So it can't be perfectly addressed, but some things leading to it seem like they can be found. > https://arxiv.org/abs/2401.11817 This seems like it's true since LLMs are a finite size, but in Google's case it has a "truth oracle" (the websites it's quoting)… the problem is it's a bad oracle.

Can you cite a source for this, or are you speculating? My understanding was the opposite -- that the activity of a confabulating LLM is indistinguishable from one giving factually accurate responses. https://arxiv.org/abs/2401.11817

>People keep throwing this phrase around in relation to LLMs, when not a single “fundamental limitation” has been rigorously demonstrated to exist Some limitations are not rigorously demonstrated to be fundamental, but continuously present from the first early LLMs yes. Shouldn't the burden of proof be on those who say it can be done? And some limitations are fundamental, and have been rigorously demonstrated, e.g.: https://arxiv.org/abs/2401.11817?utm_source=chatgpt.com

For LLM's no, but the explanation is wrong also, it has nothing to do with 'false memories' and has everything to do with how LLMs work. Here is the paper. https://arxiv.org/abs/2401.11817 RAG and fine tuning improve domain specificity and may reduce the problem to a level where you don't care, but it will always be there. Clean data would help reduce the incidents, possibly to a level that is more usable, but also don't remove the problem. Considering next token prediction as serial runs on multi tape TMs, with the previous output as the input can help. Especially if you consider Microsoft's 1.53bitnet that requires full precision for training but can reduce weights to just the sign components for inference. Unfortunately all paths to explain this I have require graduate level complexity theory and/or diff geometry. Or you relive the Brouwer–Hilbert controversy by trying the logic path.

Something among the lines of : "No matter how many resources you put into an LLM, it will always hallucinate". And researchers proven that : https://arxiv.org/abs/2401.11817 LLMs are just statistical models and just "regurgitate" what they've been trained by. They have no will, no desires, don't feel anger, hunger, joy, nothing. How can you you say LLM can match the cat intelligence "without too much trouble" with "heavy-retraining" ? This is a just some "alignement". Please don't overestimate LLMs as they just mimic just a few of visible human behaviour with some statistical tricks. You just need to show a dog once to a cat so it understands what a dog is, how many times do you need to show it to an LLM to understand what is a dog ?

I'm not overthinking it, some papers describe it [0] and [1] for example. I agree for some subjects you can deal with some error. The problem is not how often but how bad just one single error can be. My point was on controversial topic, where a single error can deal serious damage. Yes it must be error-free like for a nuclear reactor. Just imagine a Q&A chatbot answering questions on the subject of Israel and Palestine or something else really touchy, do you really think you can afford any error/hallucination ? [0] : https://arxiv.org/abs/2409.05746 [1] : https://arxiv.org/abs/2401.11817

Since the first day I consider AI as a noob working very fast and always making errors. My main usage is for tedious and non-critical tasks such as boilerplates. And I always review what has been generated like a merge/pull request, because in a sense that's what it is. What reinforces my opinion are papers like these : https://arxiv.org/abs/2409.05746 https://arxiv.org/abs/2401.11817