Technical Report on the Pangram AI-Generated Text Classifier

"Text is simply not information dense enough to be able to decode some arbitrary signal of provenance from it...it's a bad fiction to perpetuate that any of this is anything more than tarot card reading." Not true at all. Pangram is highly effective and has a very low false positive rate. The post here is impressive for a small project, it looks like they independently thought of one of the core ideas Pangram uses of creating twins to compare. You can see how it works here: https://arxiv.org/pdf/2402.14873

Pangram is probably the best known example of a detector with low false positives, they have a research paper here: https://arxiv.org/pdf/2402.14873. They do have an API but not sure if you need to request access for it. For humans I think it just comes down to interacting with LLMs enough to realize their quirks, but that's not really fool-proof.

I'm not affiliated with it in any way, but you might want to check out the leading model https://arxiv.org/abs/2402.14873 and the mixed authorship detection https://www.pangram.com/blog/pangram-3-0-technical. LLM detection has come a long way since the first GPTZero. Human heuristics - I've prompted millions of tokens across every frontier model iteration for all manner of writing styles and purposes - also helps greatly. Concerning to me are long-time posters who (perhaps unknowingly) advance the decline of this human community by encouraging the people breaking HN guidelines. Perhaps spending a few hours on Moltbook might help develop such a heuristic, since "someone who knows how to write" is just a Claude model with a link to the blogpost. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Thanks for your comment!

Human editors, I suspect, would pick up the "tells" of generated text, although as we know, there's a lot of false positives in that space. But it looks like Pangram is a text classifying NN is trained using a technique where they get a human to write a body of text on a subject, and then various LLMs to write a body of text on the same subject, which strikes me as a good way to approach the problem. Not that I'm in anyway qualified to properly understand ML. More details here: https://arxiv.org/pdf/2402.14873

I ran the introduction chapter through Pangram [1], which is one of the most reliable AI-generated text classifiers out there [2] (with a benchmarked accuracy of 99.85% over long-form text), and it gives high confidence for it having been AI-generated. It's also very intuitively obvious if you play a lot with LLMs. I have no problem at all reading AI-generated content if it's good, but I don't appreciate dishonesty. [1]: https://www.pangram.com/ [2]: https://arxiv.org/pdf/2402.14873

We benchmark on pre-2023 datasets of O(10M) documents not in our training set. Other detectors seem to have between 1-3% false positive rate and ours is around 1 in 10,000 as of our latest model update. We do a lot of active learning + core set selection to keep FPR low and improve recall on larger LLMs. Our white paper with some methodology is here: https://arxiv.org/abs/2402.14873