The Internal State of an LLM Knows When It's Lying

https://arxiv.org/abs/2304.13734 LLMs have an inner/internal state. Deep neural networks are weird and there is a lot going on in them that makes them very different from the state machines we're used to in binary programs.

Related: GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734 LLMs Know More Than What They Say - https://arjunbansal.substack.com/p/llms-know-more-than-what-... Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334

That would be a huge result it if works. The paper: "Our approach is to train a classifier that outputs the probability that a statement is truthful, based on the hidden layer activations of the LLM as it reads or generates the statement. Experiments demonstrate that given a set of test sentences, of which half are true and half false, our trained classifier achieves an average of 71% to 83% accuracy labeling which sentences are true versus false, depending on the LLM base model."[1] It's not entirely clear what they measured and whether what they measured is useful. Is this a major result? [1] https://arxiv.org/pdf/2304.13734

I believe the paper to be cited is "The Internal State of an LLM Knows When It's Lying", published last year: https://arxiv.org/abs/2304.13734

There are multiple papers and efforts that have inspected the internal state of LLMs. One could even see the word2vec analysis along these lines, as evidence that the model is specializing neurons One such example: The Internal State of an LLM Knows When It's Lying (https://arxiv.org/abs/2304.13734) Searching phrases like "llm interpretability" and "llm activation analysis" uncover more https://github.com/JShollaj/awesome-llm-interpretability

The drafts have to be evaluated either by a human or llm. Doing that for every request does not scale when you have millions of users. >Just do the comparison on the user’s machine if the LLM provider is that cheap. This is not possible. Users don't have the resources to run these gigantic models. LLM inference is not cheap. Open ai, Google aren't running profit on free cGPT or Bard. >P.S. Also aren’t LLMs deterministic if you set their “temperature” to zero? Are there drafts if the temperature is zero? If not, then that’s the same as removing the randomness no? It's not a problem of randomness. a temp of 0 doesn't reduce hallucinations. LLMs internally know when they are hallucinating/taking a wild guess. randomness influences how that guess manifests each time but the decision to guess was already made. https://arxiv.org/abs/2304.13734

Well it does have concept of right and wrong, as evidenced here https://chat.openai.com/share/e117e800-f579-4eec-9192-e9912f... you can ask it wrong answers only and it will oblige. Correctness is something it learns. I've read a few papers about hallucinations, and the jury is still out on whether a model knows when it's hallucinating, if we assume hallucinations are orthogonal to correctness Now this distinction isn't very useful in the grand scheme of things because in the end the output is wrong anyway, but it doesn't make asking to work along the axis of correctness cargo cult Further reads https://arxiv.org/abs/2304.13734 https://arxiv.org/abs/2305.18248

That's not true. There's been several papers probing this with different methodologies and the conclusion is pretty clear. LLMs know a whole lot more about the uncertainty of their predictions than they say. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734 LLMs Know More Than What They Say - https://arjunbansal.substack.com/p/llms-know-more-than-what-... Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334