If we approach the question as engineers, scientifically, with numbers and studies, not anecdotes and hand-waving, then Clojure is hands down the best language in terms of productivity and bug reduction. To this day, I know of no study that was able to demonstrate superiority of statically-typed languages - [1]. What studies clearly show, is that both in terms of productivity [2] and bug reduction [3], expressivity reigns supreme. And Clojure is the most expressive [4] out of languages that can leverage huge ecosystems (Java and JS, soon C++ through Jank dialect). [1] https://danluu.com/empirical-pl/ [2] PBX study from Economics of Software Quality by Caper Jones [3] https://arxiv.org/pdf/1901.10220 [4] https://redmonk.com/dberkholz/2013/03/25/programming-languag...
On the Impact of Programming Languages on Code Quality
Emery D. Berger, Celeste Hollenbeck, Petr Maj, Olga Vitek, Jan Vitek
2019 · arXiv
- HN score
- 0
- Mentions
- 10
- Readers
- 9
- Months
- 9
From the abstract
This paper is a reproduction of work by Ray et al. which claimed to have uncovered a statistically significant association between eleven programming languages and software defects in projects hosted on GitHub. First we conduct an experimental repetition, repetition is only partially successful, but it does validate one of the key claims of the original work about the association of ten programming languages with defects. Next, we conduct a complete, independent reanalysis of the data and statistical modeling steps of the original study. We uncover a number of flaws that undermine the conclusions of the original study as only four languages are found to have a statistically significant association with defects, and even for those the effect size is exceedingly small. We conclude with some additional sources of bias that should be investigated in follow up work and a few best practice recommendations for similar efforts.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
10 selected excerpts · 10 total mentions
> all methodologies have tradeoff's go and clojure sit near the bottom of this list in bugs/committ: https://arxiv.org/pdf/1901.10220.pdf You have linked a study which, if you read the abstract, effectively claims "we tried to reproduce a study about bugs-by-programming-language, and could not. We have concluded their methodology was flawed". I can't find information in your linked source that support anything about type-system complexity resulting in more bugs. > But if go has so many problems why do people use it? If candy is bad for your health, why do people eat it? Surely people eating candy is proof it's good for your health. That's the argument you're making…
Read the full comment
here. We can show candy is bad for your health, and the comment you originally replied to provided a mix of reasons why they saw go as having problems and ignoring programming research. All of those are more evidence than what you've replied with, which is essentially "but go is popular", entirely ignoring their concrete points. Anyway, if people choosing a language is proof that a language doesn't have problems, that surely means PHP, Java, python, C, etc have fewer problems than Go since they all are used much more than it. Again, a language being good and people using it are not necessarily related. People eat candy despite it being bad for them. People code in Go despite it being designed like "C, but with GC" Perhaps rob pike said it best: > The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt As rob pike so clearly said, perhaps go is successful because it is intentionally a poorly designed language, intentionally one that eschews programming research in order to appeal to python programmers.
> Do you have any info on this? There have been some studies, and while all methodologies have tradeoff's go and clojure sit near the bottom of this list in bugs/committ: https://arxiv.org/pdf/1901.10220.pdf Which is fascinating because they are very different languages whose unifying theme is simplicity. But this actually fits well if you know another important fact: bugs occur along organizational lines. https://augustl.com/blog/2019/best_bug_predictor_is_organiza... If you've been around long enough, you realize a lot, if not most bugs, come from not understanding someone else's code. Type systems arn't going to help you, when you don't understand what the code they…
Read the full comment
are "protecting" is doing. > The parent poster wasn't saying no one uses go, but that people use go despite its systematic problems, and tbh your comment mostly doesn't argue against that thesis, just arguing that it's successful But if go has so many problems why do people use it? Javascript, bash, and go have a unifying trait. They are useful, they solve a problem. But unlike javascript and bash, go wasn't and isn't integrated anywhere. No ecosystem forced people into go. They choose it, writ large. Why? Well because languages are more than there type systems.
> I'm not sure what evidence your fact is based on, feel free to present some evidence for it. The linked meta-study in the article we're commenting on. https://danluu.com/empirical-pl/ "under the specific set of circumstances described in the studies, any effect, if it exists at all, is small" And this question has generated the most rebuttals and retractions I've ever seen for flawed studies in computer science - it's notorious. Remember this? https://dl.acm.org/doi/10.1145/3340571 > Well, this paper found that a conservative underestimate of 15% of bugs would be saved through static typing Yeah does look that one shows a larger effect and has not been rebutted or retracted. One positive measurement against many negative measurements.
There's a speculation in the first citation: > Python is not maintanable above 50k to 100k lines of code and because of that people consider this code bases very large Note that this is a speculation, not a conclusion, the article is not very thorough, and LOC is mostly used because it is convenient to measure, not because it is what we are trying to measure. I'd personally consider 50k a "medium" code base, and 10k is "small". > Worth keeping in mind a that a Python program will be about half the LOC of a Java program doing the same thing. LOC is a confounding variable. > Bugs are also proportional to lines of code... Not supported by the citation. The citation measures…
Read the full comment
bugs per line of code, and finds that for 500 kloc of code, the average number of bugs will be somewhere between zero and 25,000. That's a very wide range. To be clear, I'm not really trying to fight against Python or for Java here. I'm just giving my reasoning for why I might personally choose one or the other. I think that the idea that you would switch languages because Python is slow is actually far more situational. You might have ten reasons to choose Python or Java, runtime performance may only be one of those factors, it may not be heavily weighted, and in some cases, Python runtime performance can be extremely fast (I do a lot of NumPy stuff... it's great). The relationship between programming language and code quality is, at best, a difficult relationship to study. It's hard to make any kind of direct statement like "using language X results in more bugs than using language Y" and back it up by evidence, even though we believe it to be true. Individual statistics which relate some variable to LOC is not useful in isolation. See: https://arxiv.org/abs/1901.10220 (a reproduction of an earlier result, that invalidates many of the conclusions)
First of all, I am arguing for the base case: absent any studies, the null hypothesis is naturally that there are no major differences between programming paradigms. On the issue of bug counts, there was one huge study, that was successfully reproduced [0], which finds that there is a real but small negative correlation between FP and bug counts in large GitHub projects. For productivity I haven't been able to find any large-scale studies that compare functional vs imperative/OOP languages. I did find one small scale study [1] that found significantly better productivity for Haskell (and a Lisp dialect) vs Ada, C++ and a few others, but it was about a specific small…
Read the full comment
prototype problem with just a handful of programmers participating (the Haskell program had 85 lines, the worse, C++, had 1105; and programs took between 10h and 54h to finish - hardly representative of large-scale coding). Most other studies on productivity that I could find [2], [3] didn't include any FP languages at all. [0] https://arxiv.org/pdf/1901.10220.pdf [1] https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.36... [2] https://www.e-informatyka.pl/attach/e-Informatica_-_Volume_1... [3] https://www.researchgate.net/publication/4261726_Do_Programm...
> > it enhances the ability to reason about code > In your opinion. Correct. But my opinion is informed by years of experience with functional programming, procedural programming, and OOP (each). > Please provide any evidence that this is true, especially the ‘safe’ part. Any study I’ve read says there’s almost no effect on language choice at all on bug rate. Sure, here is a paper which provides the evidence you're looking for: "A Large Scale Study of Programming Languages and Code Quality in Github" (https://dl.acm.org/doi/10.1145/2635868.2635922). That paper concludes: > The data indicates functional languages are better than procedural languages; it suggests that…
Read the full comment
strong typing is better than weak typing; that static typing is better than dynamic; and that managed memory usage is better than unmanaged. I personally don't trust academic studies on programming language effectiveness, since they often contradict each other (e.g., https://arxiv.org/abs/1901.10220 contradicts the paper I cited above). But if you're looking for a peer-reviewed paper, there you go. Choosing the right paradigm absolutely has an effect on the bug rate. If you have to write more code, repeat code, or deal with low-level details irrelevant to the problem at hand, you are going to make more mistakes. On top of that, functional languages often have better type systems than procedural/OOP ones, which helps with bug catching that much more. Taking this to extreme, with dependent type systems you can actually verify arbitrary mathematical properties of your programs. How much experience do you have with functional programming? I have professional and academic experience with both functional programming and OOP, and everyone I've met with that experience agrees about the trade-offs I've been discussing in my comments.
Sure, however nobody has been shown that one approach is more costly than the other in practice. The most recent research available is the replication of the large scale GitHub study [1]. The relative effects of language choice on code quality is less than 1%. The main finding is that language choice hardly matters at all. On the other hand, we can easily measure the effects of factors like sleep [2], overwork [3], and happiness [4] on code quality. If static typing was an actual factor, we’d see exactly the same kinds of effects. There’s nothing wrong with enjoying static typing, but there’s simply no evidence that it plays any role past personal preference. Different…
Read the full comment
people solve problems in different ways, and have different pain points. It's entirely possible that each type discipline appeals to different mindsets. That is a value in itself. [1] https://arxiv.org/abs/1901.10220 [2] https://arxiv.org/pdf/1805.02544.pdf [3] http://web.archive.org/web/20090824001133/http://www.curt.or... [4] http://neverworkintheory.org/2014/05/01/happy-sw-devs-solve-...
> this evidence has been around for ages Evidence of what? This is not a study of software development at all, but of prototyping using a program with no more than a few hundred lines. I'm talking about effects on software development. The issue of prototyping/tiny programs is a completely separate one. > Substantiate that I don't know how I can substantiate the lack of evidence, but here's a recent failed attempt to find evidence: https://arxiv.org/pdf/1901.10220.pdf > Lack of knowledge by an HN poster is not evidence of lack of evidence. True. While I have been following the subject for at least the past 20 years (and I have read your PDF several times already,…
Read the full comment
including shortly after it was published), it is certainly possible I have missed something. If there's any evidence of a large effect you believe you've found, let me know. > Evidence please? Sure. Here's code review, for example (I don't have time to discuss each paper -- as they're quite different -- but they all paint a similar picture). BTW, while programming language studies are debating effects in the 0-15% range, these report effects in the 30-80%: * What We Have Learned About Fighting Defects, 2002 -- https://www.cs.umd.edu/~mvz/pub/eworkshop02.pdf * The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data, 2009 -- https://www.pitt.edu/~ckemerer/PSP_Data.pdf * Large-Scale Analysis of Modern Code Review Practices and Software Security in Open Source Software, 2017 -- https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-... * What Types of Defects Are Really Discovered in Code Reviews?, 2003 -- https://ieeexplore.ieee.org/document/4604671 * Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?, 2014 -- https://www.testroots.org/assets/papers/2014_beller_bacchell... * The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects, 2014 -- https://dl.acm.org/citation.cfm?id=2597076 * Best Kept Secrets of Peer Code Review, 2013 -- https://smartbear.com/SmartBear/media/pdfs/Best-Kept-Secrets... * Modern Code Review: A Case Study at Google, 2018 -- https://storage.googleapis.com/pub-tools-public-publication-... If we summarize our current state of knowledge (not myth) it is this: process matters a lot; language matters little, if at all (with all the caveats I mentioned in other comments, such as diminishing returns etc.).
That study: 1. Found a statistically significant, but a rather small effect (low single digit % IIRC) that cannot justify language/paradigm choice (choose FP for 1.5% fewer bugs!). If anything, it's evidence against a large effect. 2. Had most of even that small effect disappear on reproduction, which increases the evidence against a large effect: https://arxiv.org/abs/1901.10220
Keep following the thread.
More in Programming languagesMore from the same topics.
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Dissolving the Fermi Paradox
Anders Sandberg, Eric Drexler, Toby Ord
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Understanding deep learning requires rethinking generalization
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Talent vs Luck: the role of randomness in success and failure
A. Pluchino. A. E. Biondo, A. Rapisarda
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.