Skip to content
The methodOpen data
Back to the collection

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

Katherine Tian, Eric Mitchell, Allan Zhou

Show all 8 authors

Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher D. Manning

2023 · arXiv

Language modelsMachine learning
Read the paper
HN score
0
Mentions
20
Readers
4
Months
7
What’s in a score?

From the abstract

A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to an expert in cases of low-confidence predictions. Recent studies have shown that unsupervised pre-training produces large language models (LMs) whose conditional probabilities are remarkably well-calibrated. However, the most widely-used LMs are fine-tuned with reinforcement learning from human feedback (RLHF-LMs), and some studies have suggested that RLHF-LMs produce conditional probabilities that are very poorly calibrated. In light of this perceived weakness, we conduct a broad evaluation of methods for extracting confidence scores from RLHF-LMs. For RLHF-LMs such as ChatGPT, GPT-4, and Claude, we find that verbalized confidences emitted as output tokens are typically better-calibrated than the model's conditional probabilities on the TriviaQA, SciQ, and TruthfulQA benchmarks, often reducing the expected calibration error by a relative 50%.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

20 selected excerpts · 20 total mentions

dcx
Neutral mention · rule
Hello! I'm one of the main authors of the paper. Thanks for engaging with our work so thoughtfully – that's a very clear and valid question. We didn't get around to addressing this within the paper itself – 80 pages is a lot, and deadlines, etc. But I have unpublished experiments that show that in a reasonably broad setting I'm doing some work in, verbalized probabilities are restoring a distribution that looks almost identical to the base distribution. It is not possible to demonstrate this on frontier models, since their public models are already mode-collapsed, and they don't share the base model or logprobs anyway. But I've established this to my personal satisfaction…
Read the full comment
on large local models which offer base / post-trained pairs. To share some intuition on why one might believe this is occurring: there are a bunch of tasks implicit in the pre-training corpus that encourage the model to learn this capability. Consider sentences in news and research articles like: "Scientists discover that [doing something] increases [some outcome] on [some population] by X%". It seems quite natural that the model might learn a pathway by which it can translate its base probabilities into the equivalent numeric tokens in order to "beat" the task of reducing loss on the "X%" prediction. I can even almost visualize how this works mechanically in terms of what the upper layers of an MLP would do to learn this, i.e. translating from weights into specific token slots. And this is almost certainly more parameter-efficient than constructing an entire separate emulated reality for filling in X. Although I'm not ruling out that the latter might still be happening – perhaps some future interp research might be able to validate this! I'm actually working on a paper right now that packs up some of the above findings in passing. But if helpful in the meantime, this is also building on related work by Tian et al. 2023, "Just Ask for Calibration" [1] and Meister et al. 2024, "Benchmarking Distributional Alignment of LLMs", that give some extra confidence here. Their findings indicate that whether or not they are rooted in the model's base probabilities, they seem to be useful for the purposes that people care about. (Oh, and you can probably set up an experiment to verify this independently with vLLM in a few Claude Code requests!) Hope that was helpful – feel free to ping with follow-ups! (Although replies might be a little delayed, I happened to see this at a good time; having quite a crunchy week) [1] https://arxiv.org/abs/2305.14975 [2] https://arxiv.org/abs/2411.05403
Original comment Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity
famouswaffles
Neutral mention · rule
Related: GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734 LLMs Know More Than What They Say - https://arjunbansal.substack.com/p/llms-know-more-than-what-... Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
Original comment LLMs know more than they show: On the intrinsic representation of hallucinations
cosmojg
Neutral mention · rule
What was the point of moving away from the base model? I can't stop asking this question. Conversational formatting is achievable with careful prompting and a bit of good old-fashioned heuristic post-processing, and it was easier to achieve consistent results before RLHF took off. Now we still have to do a bunch of prompt hacking to get the results we want[1], but it's more complicated and the performance of the model has degraded significantly[2]. All the cargo culting toward agentic chatbots and away from language prediction engines might please the marketing and investor relations departments, but it's only setting us back in the long run. [1] https://arxiv.org/pdf/2310.06452.pdf [2] https://arxiv.org/pdf/2305.14975.pdf
Original comment Anthropic Claude performance depends heavily on prompt style
og_kalu
Neutral mention · rule
>I know my world model is fundamentally incomplete. Even more foundationally, I know that there is a world, and when my world model and the world disagree, the world wins. Yeah this isn't really true. There's not how humans work. For a variety of reasons, Plenty stick with their incorrect model despite the world indicating otherwise. In fact, this seems to be normal enough human behaviour. Everyone does it, for something or the other. You are no exception. And yes LLMs can in fact tell truth from fiction. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models…
Read the full comment
Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 Your argument seems to boil down to "they can't perform experiments" but that isn't true either.
Original comment I disagree with Geoff Hinton regarding "glorified autocomplete"
famouswaffles
Neutral mention · rule
Pretty much all research (and there's a fair few with different methodologies) on this converge on the same conclusion: LLMs internally know a lot more about the uncertainty and factualness of their predictions than they say. "LLMs are always hallucinating" is a popular stance but wrong all the same. Maybe rather than asking Why models hallucinate, the better question is to ask "Why not?". During pre-training, there's close to zero incentive to push any uncertainty to the forefront (words). GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear…
Read the full comment
Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734 LLMs Know More Than What They Say - https://arjunbansal.substack.com/p/llms-know-more-than-what-... Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
Original comment GPTs and Hallucination
famouswaffles
Neutral mention · rule
That's not true. There's been several papers probing this with different methodologies and the conclusion is pretty clear. LLMs know a whole lot more about the uncertainty of their predictions than they say. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734 LLMs Know More Than What They Say - https://arjunbansal.substack.com/p/llms-know-more-than-what-... Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
Original comment AI is growing faster than companies can secure it, warn industry leaders
og_kalu
Neutral mention · rule
>An LLM can't tell the difference between fact and fiction, because it can't apply logic. Not true. They can differentiate it just fine. Of course being able to tell the difference and being incentivized to communicate it are 2 different things. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824
Original comment Language models and linguistic theories beyond words
og_kalu
Neutral mention · rule
There really is no difference. Either there is utility or there isn't. "Fake" intelligence that produces results is just something that does not make any sense. LLMs may be worse but Humans confidently state something they are uncertain about all the time lol. Maybe not experts in general but then that's still comparing LLMs to a small percentage of humanity. At any rate, Unlike many seem to think, The issue is not in fact the lack of ability to distinguish truth and fact from fiction. Turns out being able to distinguish the two and having the incentive to communicate that are 2 different things. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for…
Read the full comment
Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824
Original comment Role play with large language models – Nature
og_kalu
Neutral mention · rule
>The human model isn’t just trained in language though, it’s trained on the world. It's not trained on "the world". It's trained on a small slice of it, a few senses that are themselves slimmed down and then fabricated at parts. To the bird that can feel and sense electromagnetic waves intuitively to guide travels, you're not trained on "the world" >This is why they are so prone to hallucination. No it's not. LLMs can distinguish between truth and hallucination. They just don't care to communicate that. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models…
Read the full comment
Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824
Original comment Rats have an imagination, new research suggests
og_kalu
Neutral mention · rule
>More importantly, this is because it doesn't understand the meaning of what it generates, just juggles weights. Yeah that's not the reason at all. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824
Original comment Ask HN: What is your standard for judging when AGI exists?
og_kalu
Neutral mention · rule
>But the results are indicative of what it is, and they fall short even if your only criteria is usefulness There isn't a testable definition of general intelligence that GPT-4 fails that a chunk of humans also wouldn't. If they fall short then so do people. Guess those people aren't generally intelligent then according to you. >It can't tell the difference between truth and falsity Yes, it very much can. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their…
Read the full comment
Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 >The more interesting question is why are people like you personally offended I'm not offended about anything.
Original comment Ask HN: What is your standard for judging when AGI exists?
og_kalu
Neutral mention · rule
Related GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
Original comment The Geometry of Truth: Do LLM's Know True and False
og_kalu
Neutral mention · rule
Language models (the computation) "care" about making "correct" predictions (correct of course being determined by the data). This is why the simplest solution to reducing hallucinations is just making them more competent/better predictors (more neurons, more data). If the prediction can benefit from all the knowledge/reasoning it has accumulated from training, it'll tap that first. One of the biggest tells of hallucinations is if the model is saying something completely different each generation vs being consistent. What they just don't care about is communicating being out of distribution or making whack predictions. This becomes a problem for humans because they're…
Read the full comment
perfectly fine making things up when the above fail. But by all accounts, they do learn to distinguish these things. The computation is very much aware when it is going way off base. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824
Original comment "Hallucinating" AIs Sound Creative, but Let's Not Celebrate Being Wrong
og_kalu
Neutral mention · rule
Related - GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
Original comment The Geometry of Truth: Emergent Linear Structure in How LLMs Represent Truth
og_kalu
Neutral mention · rule
>You are unable to. This is just wrong lol. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
Original comment ChatGPT and other AI tools could disrupt scientific publishing
og_kalu
Neutral mention · rule
> Numbers (LLMs still can't do math and the ones that "can" are very brute forced and don't generalize) These seem to generalize. https://arxiv.org/abs/2211.09066 https://arxiv.org/abs/2308.00304 > Commutative properties (The (idk why anyone was surprised) "reversal curse": https://arxiv.org/abs/2309.12288) This was a retrieval from training issue not an inference one. and make no mistake, people will fail these sort of answers too (perhaps not to the same extent). Very often someone will learn a fact in a certain why and fail to recall that when asked in a reverse or roundabout way. Communicative properties are not something that occur in the vast majority of text or…
Read the full comment
language. It's not "generalization" for training to treat it as such lol. Since they can answer such questions just fine during inference, It makes little sense to say an LLM doesn't understand communicative properties. > How to say "I don't know" There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this. Don't blame the LLM here. Blame the very human data that doesn't encourage this. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 I'm not trying to say that LLMs are perfect. and as long as people understand that saying an LLM doesn't understand x doesn't mean an LLM doesn't understand anything at all, i'll happily state weaknesses.
Original comment LLMs can't self-correct in reasoning tasks, DeepMind study finds
og_kalu
Neutral mention · rule
There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
Original comment LLMs can't self-correct in reasoning tasks, DeepMind study finds
og_kalu
Neutral mention · rule
>As a rule, chatbots today have a propensity to confidently make stuff up, or, as some researchers say, “hallucinate.” At the root of these hallucinations is an inability to introspect: the A.I. doesn’t know what it does and doesn’t know. The last bit doesn't seem to be true. There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their…
Read the full comment
Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 Also even if we're strictly talking about text, there is still a ton of data left to train on. We've just barely reached what is easily scrapable online and are nowhere near a real limit yet. And of course, you can just train more than one epoch. That said, it's very clear quality data is far more helpful than sheer quantity and sheer quantity is more likely than not to derail progress.
Original comment How Will AI Learn Next?
og_kalu
Neutral mention · rule
No that's just a training process that goes, "people like this sort of responses better, try to emulate it" by feeding specific input into the training process. >So how do we get LLMs to admit when they don't know something rather than confabulate? The prompts here demonstrably help https://arxiv.org/abs/2305.14975 But otherwise nobody is certain how to do it. The simplest solution is just making them better. Hallucination is something that happens when knowledge/memory fail. By making them more competent, they hallucinate less. So even if they never care about communicating this, you wouldn't notice.
Original comment LLMs Confabulate Not Hallucinate
og_kalu
Neutral mention · rule
Related - GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
Original comment The Internal State of a Large Language Model Knows When Its Lying

Keep following the thread.

More in Language models

More from the same topics.