People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text

significantly outperforming most commercial and open-source detectors we evaluated even in the presence of evasion tactics like paraphrasing and humanization. Qualitative analysis of the experts’ free-form explanations shows that while they rely heavily on specific lexical clues, they also pick up on more complex phenomena within the text that are challenging to assess for automatic detectors. [0] Like the paper says, it's easy to point to specific clues in ai generated text, like the overuse of em dashes, overuse of inline lists, unusual emoji usage, tile case, frequent use of specific vocab, the rule of three, negative parallelisms, elegant variation, false ranges etc. But harder to articulate and perhaps more important to recognition is overall flow, sentence structure and length, and various stylistic choices that scream AI. Also worth noting that the author never actually stated that they did not use generative AI for this article. Saying that their hands were on the keyboard or that they reworked sentences and got feedback from coworkers doesn't mean AI wasn't used. That they haven't straight up said "No AI was used to write this article" is another indication. 0: https://arxiv.org/html/2501.15654v2

generated text that has received minimal or no prompting to "humanize" the writing style, not AI generated text in general. If you would like to perform a higher-quality study with more recent models, feel free (it's only fair that I ask you to do an unreasonable amount of work here given that your argument appears to be that if I don't quit my lucrative programming job and go manually classify text for pennies on the dollar, it proves that it can't be done). The reason this isn't offered as a service is because it makes no economic sense to do so using humans, not because it's impossible as you claim. This kind of "human" detection mechanism does not scale the way generation does. The cues that I rely on are also pretty easy to eliminate if you know someone is looking for them. This means that heuristics do not work reliably against someone actively trying to avoid human detection, or a human deliberately trying to sound like an LLM (I feel the need to reiterate this as many of the counterarguments to what I'm saying are to claims of this form). > I’m not going to write another detailed explanation of why your “slop === AI” premise is flawed. This isn't a claim that I made. I believe that text written with LLM assistance is not necessarily slop, and that slop is not necessarily AI generated. The only assertion I made regarding slop is that being written with LLM assistance with minimal prompting or editing is a strong predictor of slop, and that the heuristics I'm using (if present in large quantities) are a strong predictor of an article being written with LLM assistance with minimal prompting or editing. i.e. I, I am asserting that these kinds of heuristics work pretty well on articles generated by people who don't realize (or care) that there are LLM "tells" all over their work. The fact that many of the articles posted to HN are being accused of being LLM generated could certainly indicate that this is all just a massive witch hunt, but given the acknowledged popularity of ChatGPT among the general population and the fact that experts can pretty easily identify non-humanized articles, I think "a lot of people are using LLMs in the process of generating their blog posts, and some sizable fraction of those people didn't edit the output very much" is an equally compelling hypothesis.