Neutral mention · rule
In fact it came out as the absolute best in a comparison by Tencent (1). See table 4. It’s more than a year old though. 1. https://arxiv.org/pdf/2508.09101
Back to the collection
AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
Jason Chou, Ao Liu, Yuchi Deng
Show all 16 authors
Zhiying Zeng, Tao Zhang, Haotian Zhu, Jianwei Cai, Yue Mao, Chenchen Zhang, Lingyun Tan, Ziyan Xu, Bohui Zhai, Hengyi Liu, Speed Zhu, Wiggin Zhou, Fengzong Lian
2025 · arXiv
- HN score
- 0
- Mentions
- 6
- Readers
- 6
- Months
- 4
From the abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, with code generation emerging as a key area of focus. While numerous benchmarks have been proposed to evaluate their code generation abilities, these benchmarks face several critical limitations. First, they often rely on manual annotations, which are time-consuming and difficult to scale across different programming languages and problem complexities. Second, most existing benchmarks focus primarily on Python, while the few multilingual benchmarks suffer from limited difficulty and uneven language distribution. To address these challenges, we propose AutoCodeGen, an automated method for generating high-difficulty multilingual code generation datasets without manual annotations. AutoCodeGen ensures the correctness and completeness of test cases by generating test inputs with LLMs and obtaining test outputs through a multilingual sandbox, while achieving high data quality through reverse-order problem generation and multiple filtering steps. Using this novel method, we introduce AutoCodeBench, a large-scale code generation benchmark comprising 3,920 problems evenly distributed across 20 programming languages. It is specifically designed to evaluate LLMs on challenging, diverse, and practical multilingual tasks. We evaluate over 30 leading open-source and proprietary LLMs on AutoCodeBench and its simplified version AutoCodeBench-Lite. The results show that even the most advanced LLMs struggle with the complexity, diversity, and multilingual nature of these tasks. Besides, we introduce AutoCodeBench-Complete, specifically designed for base models to assess their few-shot code generation capabilities. We hope the AutoCodeBench series will serve as a valuable resource and inspire the community to focus on more challenging and practical multilingual code generation scenarios.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
6 selected excerpts · 6 total mentions
Neutral mention · rule
I like Rust and I'm an advocate of Rust, but this really isn't true (at least it hasn't been and I doubt much has significantly changed). The syntax complexity and the ecosystem haven't been ideal for LLM development. And there have been publications on findings of LLM efficacy with different languages. Rust is most often towards the lower end of efficiency/correctness when benchmarked. https://arxiv.org/html/2508.09101v1
Neutral mention · rule
In this paper AutoCodeBench https://arxiv.org/pdf/2508.09101 LLMs seemed to do best on the strongly typed and functional language Elixir. This is surprising since there isn't very much training data. However the examples that it's seen are usually high quality and there aren't a large number of different way of doing the same thing in Elixir.
Neutral mention · rule
https://arxiv.org/pdf/2508.09101 tldr 2% average point lost on Rust compared to python, gap vary by model, go has a better upper bound but opus had it 3% below python. benchmark is a bit old but research on why is there, article is just vibes
Neutral mention · rule
> The language they do best with is the one with the largest corpus in the training set. Not the case, strangely. They do best with Elixir. https://arxiv.org/pdf/2508.09101
Neutral mention · rule
https://arxiv.org/abs/2508.09101 In this benchmark, models can correctly solve Rust problems 61% on first pass — A far cry from other languages such as C# (88%) or Elixir (a “buggy dynamic language”) where they perform best (97%). I wonder why that is, it’s quite surprising. Obviously details of their benchmark design matter, but this study doesn’t support your claims.
Keep following the thread.
More in Language modelsMore from the same topics.
2023arXiv
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
60 mentions14 months
3.5HN score
2015arXiv
A Critique of the CAP Theorem
Martin Kleppmann
The CAP Theorem is a frequently cited impossibility result in distributed systems, especially among NoSQL distributed databases.
6 mentions6 months
3.5HN score
2020arXiv
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
83 mentions35 months
3HN score
2020arXiv
Copy-and-Patch Compilation: A fast compilation algorithm for high-level languages and bytecode
Haoran Xu, Fredrik Kjolstad
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.
10 mentions9 months
2.5HN score