Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

Yes, there's no physical reason it couldn't be fixed. However, I think it's less a matter of computing power, and more one of architecture and training. The inference-time code could be changed to keep a log of which neurons were activated the most when it generated a token. But then, how would we train it to use that log and report it accurately? Training on Internet documents isn't enough. Chain-of-thought reasoning helps as a workaround. However, beware that even then, the justifications aren't a reliable explanation. There's an interesting paper on this: Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting https://arxiv.org/abs/2305.04388

"major architectural decisions don't get documented anywhere" "commit messages give no "why"" This is so far outside of common industry practices that I don't think your sentiment generalizes. Or perhaps your expectation of what should go in a single commit message is different from the rest of us... LLMs, especially those with reasoning chains, are notoriously bad at explaining their thought process. This isn't vibes, it is empiricism: https://arxiv.org/abs/2305.04388 If you are genuinely working somewhere where the people around you are worse than LLMs at explaining and documenting their thought process, I would looking elsewhere. Can't imagine that is good for one's own development (or sanity).

This is interesting, but I wonder how reliable this type of monitoring is really going to be in the long run. There are fairly strong indications that CoT adherence can be trained out of models, and there's already research showing that they won't always reveal their thought process in certain topics. See: https://arxiv.org/pdf/2305.04388 On a related note, if anyone here is also reading a lot of papers to keep up with AI safety, what tools have been helpful for you? I'm building https://openpaper.ai to help me read papers more effectively without losing accuracy, and looking for more feature tuning. It's also open source :)

Recent paper from Anthropic on the fact that the reasoning output does not necessarily reflect what the LLM is "thinking". https://arxiv.org/abs/2305.04388 That being said, your idea is not unreasonable. The way DeepSeek phrased it, it just sounds like implementing such solutions might be a hassle greatly increasing complexity, and they were just focused on making an RL baseline work at scale.

In section 2 they briefly mention studies such as [1] that point out that the token outputs of a chain of thought aren't always entirely faithful to the responses of the models I'm not sure whether it wouldn't be more reliable to let the model run on latents and try to train a separate latent-reading explainer module that has at least some approximation of what we want as an explicit optimization objective. Assuming it actually is or has the potential to be better than CoT, from what I gathered from the paper the current results are mostly just more efficient token-wise. [1] https://arxiv.org/abs/2305.04388