Skip to content
The methodOpen data
Back to the collection

Do Language Models Know When They're Hallucinating References?

Ayush Agrawal, Mirac Suzgun, Lester Mackey, Adam Tauman Kalai

2023 · arXiv

Language modelsDistributed systems
Read the paper
HN score
0
Mentions
4
Readers
4
Months
4
What’s in a score?

From the abstract

State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. Such inaccurate outputs not only undermine the reliability of these models but also limit their use and raise serious concerns about misinformation and propaganda. In this work, we focus on hallucinated book and article references and present them as the "model organism" of language model hallucination research, due to their frequent and easy-to-discern nature. We posit that if a language model cites a particular reference in its output, then it should ideally possess sufficient information about its authors and content, among other relevant details. Using this basic insight, we illustrate that one can identify hallucinated references without ever consulting any external resources, by asking a set of direct or indirect queries to the language model about the references. These queries can be considered as "consistency checks." Our findings highlight that while LMs, including GPT-4, often produce inconsistent author lists for hallucinated references, they also often accurately recall the authors of real references. In this sense, the LM can be said to "know" when it is hallucinating references. Furthermore, these findings show how hallucinated references can be dissected to shed light on their nature. Replication code and results can be found at https://github.com/microsoft/hallucinated-references.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

cornel_io
Neutral mention · rule
There are various results that suggest that LLMs do internally have everything they'd need to know that they're hallucinating/wrong: https://arxiv.org/abs/2402.09733 https://arxiv.org/abs/2305.18248 https://www.ox.ac.uk/news/2024-06-20-major-research-hallucin... So I don't think it's that they have no concept of correctness, they do, but it's not strong enough. We're probably just not training them in ways that optimize for that over other desirable qualities, at least aggressively enough. It's also clear to anyone who has used many different models over the years that the amount of hallucination goes down as the models get better, even without any special attention being…
Read the full comment
(apparently) paid to that problem. GPT 3.5 was REALLY bad about this stuff, but 4o and o1 are at least mediocre. So it may be that it's just one of the tougher things for a model to figure out, even if it's possible with massive capacity and compute. But I'd say it's very clear that we're not in the world Gary Marcus wishes we were in, where there's some hard and fundamental limitation that keeps a transformer network from having the capability to be more truthful as a it gets better; rather, like all aspects, we just aren't as far along as we'd prefer.
Original comment AI Blindspots – Blindspots in LLMs I've noticed while AI coding
og_kalu
Neutral mention · rule
https://arxiv.org/abs/2305.18248 "In particular, we find that LMs often hallucinate differing authors of hallucinated references when queried in independent sessions, while consistently identify authors of real references. This suggests that the hallucination may be more a generation issue than inherent to current training techniques or representation." https://arxiv.org/abs/2303.08896 "SelfCheckGPT leverages the simple idea that if a LLM has knowledge of a given concept, sampled responses are likely to be similar and contain consistent facts. However, for hallucinated facts, stochastically sampled responses are likely to diverge and contradict one another."
Original comment ChatGPT can now see, hear, and speak – openai.com
avereveard
Neutral mention · rule
Well it does have concept of right and wrong, as evidenced here https://chat.openai.com/share/e117e800-f579-4eec-9192-e9912f... you can ask it wrong answers only and it will oblige. Correctness is something it learns. I've read a few papers about hallucinations, and the jury is still out on whether a model knows when it's hallucinating, if we assume hallucinations are orthogonal to correctness Now this distinction isn't very useful in the grand scheme of things because in the end the output is wrong anyway, but it doesn't make asking to work along the axis of correctness cargo cult Further reads https://arxiv.org/abs/2304.13734 https://arxiv.org/abs/2305.18248
Original comment How to Get ChatGPT to Stop Apologizing?

Keep following the thread.

More in Language models

More from the same topics.