Skip to content
The methodOpen data
Back to the collection

Reading Race: AI Recognises Patient's Racial Identity In Medical Images

Imon Banerjee, Ananth Reddy Bhimireddy, John L. Burns

Show all 23 authors

Leo Anthony Celi, Li-Ching Chen, Ramon Correa, Natalie Dullerud, Marzyeh Ghassemi, Shih-Cheng Huang, Po-Chih Kuo, Matthew P Lungren, Lyle Palmer, Brandon J Price, Saptarshi Purkayastha, Ayis Pyrros, Luke Oakden-Rayner, Chima Okechukwu, Laleh Seyyed-Kalantari, Hari Trivedi, Ryan Wang, Zachary Zaiman, Haoran Zhang, Judy W Gichoya

2021 · arXiv

Biology & medicineMachine learning
Read the paper
HN score
0
Mentions
5
Readers
5
Months
3
What’s in a score?

From the abstract

Background: In medical imaging, prior studies have demonstrated disparate AI performance by race, yet there is no known correlation for race on medical imaging that would be obvious to the human expert interpreting the images. Methods: Using private and public datasets we evaluate: A) performance quantification of deep learning models to detect race from medical images, including the ability of these models to generalize to external environments and across multiple imaging modalities, B) assessment of possible confounding anatomic and phenotype population features, such as disease distribution and body habitus as predictors of race, and C) investigation into the underlying mechanism by which AI models can recognize race. Findings: Standard deep learning models can be trained to predict race from medical images with high performance across multiple imaging modalities. Our findings hold under external validation conditions, as well as when models are optimized to perform clinically motivated tasks. We demonstrate this detection is not due to trivial proxies or imaging-related surrogate covariates for race, such as underlying disease distribution. Finally, we show that performance persists over all anatomical regions and frequency spectrum of the images suggesting that mitigation efforts will be challenging and demand further study. Interpretation: We emphasize that model ability to predict self-reported race is itself not the issue of importance. However, our findings that AI can trivially predict self-reported race -- even from corrupted, cropped, and noised medical images -- in a setting where clinical experts cannot, creates an enormous risk for all model deployments in medical imaging: if an AI model secretly used its knowledge of self-reported race to misclassify all Black patients, radiologists would not be able to tell using the same data the model has access to.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

5 selected excerpts · 5 total mentions

kthejoker2
Neutral mention · rule
A conceptually related finding that computer vsion models can effectively classify patient race from significantly noised x-ray images https://arxiv.org/abs/2107.10356 > However, our findings that AI can trivially predict self-reported race -- even from corrupted, cropped, and noised medical images -- in a setting where clinical experts cannot, creates an enormous risk for all model deployments in medical imaging: if an AI model secretly used its knowledge of self-reported race to misclassify all Black patients, radiologists would not be able to tell using the same data the model has access to. Potentially the method in OP's paper could be used as an adversarial critic or similar counterbalancing effort to eliminate "secret knowledge" when it's not desirable.
Original comment Training computer vision models on random noise instead of real images
lostlogin
Neutral mention · rule
> they found it looks at the name of the patient on the border of the image or something similar. The images were anonymised. If they AI cheated, it’s not obvious how. The paper is interesting. https://arxiv.org/abs/2107.10356
Original comment AI Recognises Race in Medical Images
OneEyedRobot
Neutral mention · rule
They always call for 'further investigation'. Looking at this: https://arxiv.org/pdf/2107.10356.pdf My general impression (no more than that) is a whole bunch of people crowding into a paper. The paper is mostly applying trivial image processing functions and seeing how some software they don't understand is responding. The main aim is pearl-clutching about 'bias' rather than any kind of understanding. God knows what they're going to do when any medical exam includes some kind of deep dive into the patient's genetics. No surprises. It's the nature of the era.
Original comment AI Recognises Race in Medical Images
hgial
Neutral mention · rule
It might be helpful for folks to look at the blog post written by one of the authors: https://lukeoakdenrayner.wordpress.com/2021/08/02/ai-has-the... or the paper itself https://arxiv.org/pdf/2107.10356.pdf I see a lot of "oh it's probably just picking up on x y z" when x, y, and z are things they explicitly checked for: 1) "It's probably just the names or other metadata" – they only gave it pixel data to train on. To control for things like metadata overlaid on the image (e.g., a name written on the image) they divided the images into 3x3 sections and trained classifiers on each section separately. 2) "It's probably some artifact of how the hospital marked up the images"…
Read the full comment
– they used something like 7 different datasets from different hospitals and different modalities (X-Ray and CT). If it is cheating somehow, it's not doing it in an obvious way that you can think of in a minute or two. Also note that they had more than just medical folks working on the paper; the author list includes plenty of computer scientists. It's unlikely they're making an elementary ML mistake here.
Original comment AI Recognises Race in Medical Images
bArray
Neutral mention · rule
Disclaimer: Worth noting that this paper is apparently pre-print [1] and was submitted to the Lancet, who are not particularly reliable in recent times (flaws hydroxychloroquine study [2], stamped out early investigation in COVID lab-leak possibility [3]). Regarding the blog and paper: > AI can trivially learn to identify the self-reported racial identity of patients to an absurdly high degree of accuracy If it is 'trivial' for a model to learn race, then what they are detecting is likely pretty obvious. In their paper they use 'private and public datasets' [4]. There is probably exactly where the problem occurs. People of the similar race tend to live in groups together,…
Read the full comment
if you collect data in some area, there's a good chance you source data from the same ethnic pool. What it appears they are detecting is the difference between different machines and practices. To test this, they should collect their own medical data on the same machine. They also need to figure out how to remove the bias of the medical staff themselves. Another assumption is that just because the medical doctors cannot see differences in the anatomy of races, it doesn't mean there isn't one. One of the commentators on this blog indicated that dark skin may affect the absorption of x-rays, for example. > AI does learn to do this when trained for clinical tasks Just because AI models can detect race from medical data, doesn't mean they will. They may in the early stages learn such simplistic relationships, but as accuracy increases, it will be forced to look for more complex classifiers/indicators. Finally, this blog really skirts around the important thing - how does this affect patient care? Is it beneficial to actually have race-based medical plans? We know that some medicines affect different races - for example in pain and pain management [5]. I think the point to be made here isn't 'racism' but the possibility of bias that people training models should be aware of. Some of this bias will have a positive outcome, some will have a negative outcome. [1] https://www.bibsonomy.org/bibtex/c3e2ede2f835d35c08c54a67095... [2] https://www.nbcnews.com/health/health-news/lancet-retracts-l... [3] https://www.washingtonexaminer.com/news/wuhan-lab-collaborat... [4] https://arxiv.org/pdf/2107.10356.pdf [5] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3654683/
Original comment AI has the worst superpower medical racism

Keep following the thread.

More in Biology & medicine

More from the same topics.