Neutral mention · rule
That's a cute story (read to the end for the twists). I really came to hate this problem when reading COVID research. Primary case for the prosecution is a paper like Flaxman et al, "Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe" [1]. It claims that lockdowns saved ~3 million lives. Amongst other problems this paper has: it is built on circular logic [2] and an over-fitted model [3], has numerous other statistical problems [4], required an enormous country-specific fudge factor and hiding data points to cover up the way Sweden's results disproved their claims, and their model always claimed that the last NPI was the most effective…
Read the full comment
regardless of what it was. The cherry on top is that it disclaims its own results, literally admitting in the text that their counterfactual model is "illustrative only" and that "in reality, even in the absence of government interventions we would expect Rt to decrease and therefore would overestimate deaths in the no-intervention model" i.e. they already knew their conclusions were wrong when they wrote it. (This rather important caveat didn't appear in their press release about the paper, of course [5]). Yet according to Google Scholar this paper has been cited nearly 3000 times in the ~three years since its publication. It gets cited several times per day. I've watched the citation count go up with morbid fascination. What are people citing it for? If we do a reverse citation search and check some papers, we see immediately that it's being cited for a wide variety of almost random statements, none of which it actually supports: 1. "Health-care workers, seniors and those with underlying health conditions are at particularly high risk" [6] [8]. This claim appears with identical wording in two different papers, but Flaxman et al don't present any data on this or even reference risk stratification by job as far as I can tell. It's certainly not the focus of the paper. 2. "However, because most countries have implemented multiple infection control measures, it is difficult to determine the relative benefit of each" [7]. Flaxman et al claim it's easy to determine the benefit of each and that they did so. 3. "health agencies have long relied on predictive models to estimate future trends and to assess the potential effectiveness of various disease control methods" [9]. The paper doesn't show anything about the history of health agency decision making. etc. Even when citations characterize its claims correctly, they are just taking its assertions at face value without realizing that the paper's methodology is circular and even the authors don't believe their own numbers. Of what use are citations in this environment? These aren't cherry picked examples, they're literally just randomly selected by when I happened to search Scholar. Even so, most of the citations are wrong. [1] https://www.nature.com/articles/s41586-020-2405-7 [2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7674856/ [3] https://www.nature.com/articles/s41586-020-3025-y [4] https://nicholaslewis.org/did-lockdowns-really-save-3-millio... [5] https://www.imperial.ac.uk/news/198074/lockdown-school-closu... [6] https://www.nature.com/articles/s41577-020-00434-6 [7] https://jamanetwork.com/journals/jama/article-abstract/27683... [8] https://www.sciencedirect.com/science/article/pii/S147330992... [9] https://journals.sagepub.com/doi/full/10.1177/00375497231152...