Cameron R. Jones, Benjamin K. Bergen
2025 · arXiv
We evaluated 4 systems (ELIZA, GPT-4o, LLaMa-3.1-405B, and GPT-4.5) in two randomised, controlled, and pre-registered Turing tests on independent populations. Participants had 5 minute conversations simultaneously with another human participant and one of these systems before judging which conversational partner they thought was human. When prompted to adopt a humanlike persona, GPT-4.5 was judged to be the human 73% of the time: significantly more often than interrogators selected the real human participant. LLaMa-3.1, with the same prompt, was judged to be the human 56% of the time -- not significantly more or less often than the humans they were being compared to -- while baseline models (ELIZA and GPT-4o) achieved win rates significantly below chance (23% and 21% respectively). The results constitute the first empirical evidence that any artificial system passes a standard three-party Turing test. The results have implications for debates about what kind of intelligence is exhibited by Large Language Models (LLMs), and the social and economic impacts these systems are likely to have.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
16 selected excerpts · 16 total mentions
https://arxiv.org/abs/2503.23674 From the abstract: "When prompted to adopt a humanlike persona, GPT-4.5 was judged to be the human 73% of the time: significantly more often than interrogators selected the real human participant. LLaMa-3.1, with the same prompt, was judged to be the human 56% of the time"
That’s misleading, because the reason Claude answers that way is almost certainly due to reinforcement learning that deliberately prevents models from claiming they’re conscious. That’s not a valid reason for saying they fail the Turing test. By most normal standards, they can definitely pass the Turing test. See e.g. https://arxiv.org/abs/2503.23674
When people judge blindly, the are more likely to think the human is the AI and the AI is the human. 73% judged GPT 4o to be the human. https://arxiv.org/abs/2503.23674 Not only are people bad at judging this, but are directionally wrong.
In the test mentioned in nearby comments (https://arxiv.org/abs/2503.23674) ELIZA only got 27% suggesting the test wasn't that easy to fool.
> nothing actually passes the Turing test Says who? I had already found this study, published almost a year ago, saying that they do: https://arxiv.org/abs/2503.23674 There doesn't seem to be a super-rigorous definition of the Turing Test, but I don't think it's reasonable to require it to fool an expert whose life depends on the correct choice. It already seems to be decently able to fool a person of average intelligence who has a basic knowledge of LLMs. I agree that we don't really have AGI yet, but I'd hope we can come up with a better definition of what it is than "we'll know it when we see it". I think it is a legitimate point that we've moved the goalposts some.
"Proper" may be doing some work here, but such a test was run last year and GPT-4.5 and LLaMa-3.1-405B both passed. Oddly, GPT-4.5 was judged as human significantly more often than chance. https://arxiv.org/abs/2503.23674
Landauer and Bellman, absolutely put ELIZA to an adversarial Turing test, and called it such, in 1999. [0] But... Over in 2025, ELIZA was once again, put to the Turing test in adversarial conditions. [1] And still had people think it was a real person, over 27% of the time. Over a quarter of the testees, thought the thing was a human. The "ELIZA Effect" wasn't coined because everyone understands that an AI isn't conscious. [0] https://books.google.com.au/books?id=jTgMIhy6YZMC&pg=PA174 [1] https://arxiv.org/html/2503.23674v1
>Not even close. "Neural networks" in code are nothing like real neurons in real biology Hence the simplified. The weights encoding learning and inteconnectedness and nonlinear activation and distributed representation of knowledge is already an approximation, even if the human architecture is different and more elaborate. Whether the omitted parts are essential or not, is debatable. “Equations of motion are nothing like real planets" either, but they capture enough to predict and model their motion. >The "turing test" doesn't exist. Turing talked about a thought experiment in the very early days of "artificial minds". It is not a real experiment. It is not a real…
singural experiment protocol, but it's a well enough defined experimental scenario which for over half a century, it was kept as the benchmark of recognition of artificial intelligence, not by laymen (lol) but by major figures in AI research as well, figures like Minsky, McCarthy and others engaged with it. That researchers haven't done Turing-test studies (taking the setup from turing and even called them that) is patently false. Including openly testing LLMs: https://aclanthology.org/2024.naacl-long.290/ https://www.pnas.org/doi/10.1073/pnas.2313925121 https://arxiv.org/pdf/2503.23674 https://arxiv.org/pdf/2407.08853 https://arxiv.org/abs/2405.08007 https://www.sciencedirect.com/science/article/pii/S295016282...
> the bar for the test gets raised and eventually humans won’t pass the test themselves. At this point LLMs usually beat humans at the Turing Test! People are more likely to pick the LLM as the human, rather than the human. https://arxiv.org/abs/2503.23674
"Someone" literally did that (+/- 2 years): https://link.springer.com/book/10.1007/978-3-540-68677-4 I think it was supposed to be a more useful term than the earlier and more common "Strong AI". With regards to strong AI, there was a widely accepted definition - i.e. passing the Turing Test - and we are way past that point already: ( see https://arxiv.org/pdf/2503.23674 )
Are you asking if they fool me, or if I believe they fool people in general? The former, how would I tell? I'd have to find someone whose writing style I don't know to be the human comparison, in order to blind myself. The latter: https://arxiv.org/abs/2503.23674
>You are not just seeing consciousness in me; your brain is generating the feeling of another’s consciousness as the best explanation for the patterns you’re interacting with. It's ironic to see the most mundane and likely best answer to the problem from the model itself, while the author is getting increasingly lost in philosophical conundrums. Consciousness has no scientific definition. The only way something, anything can be conscious, is if a human that we also consider "conscious" calls it that. You could argue that's what the Turing test evaluates, but some of the most recent models have actually passed this test [1]. So where do we go from here if we're not…
convinced yet? The answer is: nowhere. Humans used to deny that animals could have consciousness because they don't have souls or aren't chosen by god according to some sacred books or something along those lines. They even used to deny that other humans have consciousness to promote slavery and slaughtering. Today many would still deny consciousness in computers even when faced with overwhelming evidence, because they might fear for their jobs and thereby their wealth and social standing. Artificial intelligence is a direct threat to the foundations of personality in a capitalist society. Because what are you still worth if you lose in every metric to a computer? Consciousness is kind of a last straw that many people will cling to for the foreseeable future when all else is gone. But that also means these discussions are utterly meaningless and only serve to promote certain world views. It's best not to twist your head about it and just accept that humans are not the pinnacle (or the end) of intelligent thought in the universe. That is the only reality I'm willing to bet on. [1] https://arxiv.org/abs/2503.23674
> With all due respect, your claim is ridiculous. It’s incredibly easy to spot an LLM in an interactive chat. The tells are endless. https://arxiv.org/abs/2503.23674?hl=en-GB "When prompted to adopt a humanlike persona, GPT-4.5 was judged to be the human 73% of the time: significantly more often than interrogators selected the real human participant" My "ridiculous" claim is confirmed by science.
I thought it was funny that in the Cameron R. Jones attempt as doing the test, 75% of judges thought GPT-4o was the human rather than the actual human. I think it illustrates both the limits of the test and that LLMs are getting quite good. (paper https://arxiv.org/abs/2503.23674) I think if you are having to accuse the humans of woeful typing and being smartphone gen fools you are kind scoring one for the LLM. In the Turing test they were only supposed to match an average human.
Now that I have a little more time to search around, I easily found this study, published March 31st this year, so not quite 3 months ago: https://arxiv.org/abs/2503.23674 I only skimmed it, but I don't see anything clearly wrong about it. According to their results, GPT-4.5 with what they term a "persona" prompt does in fact pass a standard that seems to me at least a little harder than what you said - actively picks the AI as the human, which seems stricter to me than being "unable to distinguish". It is a little surprising to me that only that one LLM actually "passed" their test, versus several others performing somewhat worse. Though it's also not clear exactly how long ago the actual tests were done - this stuff moves super fast.
You're right about interactivity, something that I overlooked -- but I think it's nevertheless the case that a large fraction of human interrogators could not distinguish a human from a suitably-system-prompted ChatGPT even over the course of an interactive discussion. ChatGPT 4.5 was judged to be the human 73% of the time in this RCT study, where human interrogators had 5-minute conversations with a human and an LLM: https://arxiv.org/pdf/2503.23674
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
A. Pluchino. A. E. Biondo, A. Rapisarda
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.
David Manheim, Scott Garrabrant
There are several distinct failure modes for overoptimization of systems on the basis of metrics.