Skip to content
The methodOpen data
Back to the collection

LLMs Get Lost In Multi-Turn Conversation

Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, Jennifer Neville

2025 · arXiv

Language models
Read the paper
HN score
0
Mentions
3
Readers
3
Months
3
What’s in a score?

From the abstract

Large Language Models (LLMs) are conversational interfaces. As such, LLMs have the potential to assist their users not only when they can fully specify the task at hand, but also to help them define, explore, and refine what they need through multi-turn conversational exchange. Although analysis of LLM conversation logs has confirmed that underspecification occurs frequently in user instructions, LLM evaluation has predominantly focused on the single-turn, fully-specified instruction setting. In this work, we perform large-scale simulation experiments to compare LLM performance in single- and multi-turn settings. Our experiments confirm that all the top open- and closed-weight LLMs we test exhibit significantly lower performance in multi-turn conversations than single-turn, with an average drop of 39% across six generation tasks. Analysis of 200,000+ simulated conversations decomposes the performance degradation into two components: a minor loss in aptitude and a significant increase in unreliability. We find that LLMs often make assumptions in early turns and prematurely attempt to generate final solutions, on which they overly rely. In simpler terms, we discover that *when LLMs take a wrong turn in a conversation, they get lost and do not recover*.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

3 selected excerpts · 3 total mentions

mrcoders
Neutral mention · rule
Looking at this thread, excited to see the community articulating what many of us have been experiencing. The shift from "prompt engineering" to "context engineering" really captures what's happening. The technical stuff (RAG, vector databases) is getting commoditized. But there's this foundational knowledge organization layer that's becoming critical. Most companies have context scattered across wikis, Slack, Google Docs. You can build sophisticated retrieval systems, but if you're feeding them fragmented information, you're missing huge optimization opportunities. There's research backing this - Microsoft/Salesforce found 39% accuracy drop in multi-turn conversations - so for the agent interaction's it's even more cirtical to give 'just enough' context (https://arxiv.org/pdf/2505.06120). When (business) context is properly structured upfront, you minimize those patterns.
Original comment The new skill in AI is not prompting, it's context engineering
hbbio
Neutral mention · rule
That's why in practice you need more than this simple loop! Pretty much WIP, but I am experimenting with simple sequence-based workflows that are designed to frequently reset the conversation [2] This goes well with Microsoft paper "LLMs Get Lost In Multi-Turn Conversation " that was published Friday [1]. - [1]: https://arxiv.org/abs/2505.06120 - [2]: https://github.com/hbbio/nanoagent/blob/main/src/workflow.ts
Original comment The unreasonable effectiveness of an LLM agent loop with tool use

Keep following the thread.

More in Language models

More from the same topics.