An AI model to predict kidney damage, trained on data from veterans, works less well in women

The study was a page-turner: Researchers at Google showed that an artificial intelligence system could predict acute kidney injury, a common killer of hospitalized patients, up to 48 hours in advance.

The results were so promising that the Department of Veterans Affairs, which supplied de-identified patient data to help build the AI, said in 2019 that it would immediately start work to bring it to the bedside.

But a new study shows how treacherous that journey can be. Researchers found that a replica of the AI system, trained on a predominantly male population of veterans, does not perform nearly as well on women. Their study, published recently in the journal Nature, reports that a model built to approximate Google’s AI overestimated the risk for women in certain circumstances and was less accurate in predicting the condition for women overall.

“If we have this problem, then half the population won’t benefit,” said Jie Cao, a Ph.D. student at the University of Michigan and the lead author of the paper. She said the results reinforce the need to train models on diverse groups of patients and test them on local populations, where demographic differences among patients and variations in how health care is delivered may undermine an AI system’s accuracy.

None of this is necessarily news to Google, which flagged performance issues in women in its initial paper and emphasized the need for additional testing. But the Michigan paper quantifies the extent of the problem at various stages of kidney injury, and goes further to point out how challenging the problem is to fix.

When the researchers retrained their model on sex-balanced data from Michigan and VA facilities, it performed better on women in the Michigan data but continued to struggle within the VA. That suggests its accuracy problem may be tied to issues more complicated than limited exposure to female patients with kidney damage, such as differences in treatment practices within VA facilities.

Acute kidney injury kills about 1.7 million people around the world annually. It is difficult to recognize and often causes patients to deteriorate rapidly, before lifesaving treatment can be delivered. Within the VA, 28% of patients who develop the condition die within one year, and about 6% die in the hospital, according to a recent study.

All of which makes it an alluring use case for AI.

By analyzing patient data, Google’s system, developed by the Alphabet research unit DeepMind Health, which has since been merged with Google’s health division, could tell clinicians which patients would develop the condition up to two days in advance.