An AI model predicting acute kidney injury works, but not without some tweaking

The model identified AKI 48 hours in advance, allowing ample time for clinicians to intervene and provide treatment.

10:39 AM

Author | Clarissa Piatek

scientist examining a kidney
Getty Images

In 2019, Google AI subsidiary DeepMind used a large dataset of patient records from the Veterans Health Administration to develop a predictive model for acute kidney injury—a potentially fatal condition whose prognosis improves the earlier a treatment intervention is administered. The DeepMind model purported to predict AKI 48 hours in advance, allowing ample lead time for clinicians to intervene and administer treatment.

The study team reviewed electronic health record data over a five-year period from more than 700,000 individuals.

“This is a phenomenal model because it can predict AKI up to 48 hours in advance, in a continuous manner, and has the best model performance compared to all previously published models,” said Jie Cao, a Ph.D., student in the Department of Computational Medicine and Bioinformatics. Cao is a researcher in the ML4LHS Lab, run by Karandeep Singh, M.D., MMSc., assistant professor in the Department of Learning Health Sciences and the Biomedical & Clinical Informatics Lab run by Kayvan Najarian, Ph.D., professor in the Department of Computational Medicine and Bioinformatics. All are of the University of Michigan. However, “concerns were raised about the generalizability of a model like this given the predominantly male [VA] population that it was trained on,” Cao added.

This led Cao and her colleagues to evaluate the model’s generalizability in a non-VA, more sex-balanced population. Their findings have been published in Nature Machine Intelligence. 

The researchers reconstructed aspects of the DeepMind model, then trained and validated this model on two cohorts: one comprising 278,813 VA hospitalizations (from 118 VA hospitals) and the other 165,359 U-M hospitalizations. Not surprisingly, given the 94% male population with which the original model was developed, the reconstructed model performed worse for female patients in both cohorts.

To mitigate the model’s sex-based discrepancies, researchers updated the model with data from U-M’s more sex-balanced patient population, which extended the original model from 160 decision trees to 170. This small extension improved performance in the U-M cohort both overall and between sexes.

Like Podcasts? Add the Michigan Medicine News Break on Spotify, Apple Podcasts or anywhere you listen to podcasts.

“The extended model was successful at U-M. It used the VA model as the backbone, added information from U-M, and the final product worked well for the U-M patient population,” said Cao, lead author of the paper. “When researchers would like to benefit from the rich information contained in the original model and do not want to build a new local model from scratch, our study is a good example of how ‘fine-tuning’ could work when the original model was not trained on a diverse population,” she explained.

When the extended model was applied to VA patients, however, the discrepancies in model performance between males and females actually worsened.

“This finding surprised us to some extent,” said Cao, “but it is also reasonable and helps us understand the problem better. Difference in patient characteristics is one common factor contributing to model performance discrepancy. By matching the female patients at two different health systems and still finding discrepant model performance, we actually show that difference in patient characteristics wasn’t the only reason contributing to model performance discrepancy.”

Lower performance of the extended model in female VA patients, then, was not a factor of patient characteristics or low sample size, but likely attributable to variables such as differences in practice patterns between males and females at the VA.

Overall, the study demonstrated the value of updating existing models with data from the population to which the model will be applied.  

“If a predictive model is to be taken out of one healthcare system and applied to another, the population the model was trained on is often different from the population it is going to be applied to. Even if the training population is diverse, we could observe a drop in model performance if nothing is done,” said Cao. “Our ‘extended model’ approach is to provide a solution to partially address this issue.”

To achieve peak performance, a model would, in theory, be applied only to a population matching the population it was trained on. But this is often not the case in practice.

“In the real world,” said Cao, this approach is “infeasible due to limited resources, time, expertise, etc. Our ‘extended model’ strategy is a workaround in these scenarios.”

This research is significant, said Cao, because it shows “the complexity of discrepancies in model performance in subgroups that cannot be explained simply on the basis of sample size.” It also offers “a potential strategy to mitigate the generalizability issue,” she said, and, finally, it demonstrates “the importance of reproducing and evaluating artificial intelligence studies.”

Live your healthiest life: Get tips from top experts weekly. Subscribe to the Michigan Health blog newsletter

Headlines from the frontlines: The power of scientific discovery harnessed and delivered to your inbox every week. Subscribe to the Michigan Health Lab blog newsletter


More Articles About:

All Research Topics Kidney Disease Kidney Failure Preventative health and wellness Future Think Hospitals & Centers Health Care Quality Health Tech
Health Lab word mark overlaying blue cells

Health Lab

Explore thousands of health news & research stories by visiting the Health Lab homepage for more.

Media Contact

University Hospital at U-M Health in the spring with flowering trees in foreground and Survival Flight helicopter visible

Public Relations

Department of Communication at Michigan Medicine

[email protected]

734-764-2220

Stay Informed

Want top health & research news weekly? Sign up for Health Lab’s newsletters today!

Subscribe

Featured News & Stories

Mother sits on a couch with a laptop on her lap while an elementary school age boy sits next to her playing on a tablet device.
Health Lab

New questionnaire aims to improve understanding of demand avoidance in children, teens

Pathological demand avoidance is a pattern of intense distress and avoidance of demands in children. A new questionnaire from University of Michigan researchers creates a structured way to assess these behaviors.
Blue round cells in three pictures showing how they change under stress.
Health Lab

Timing is of the essence for RNA during stress

Key determiners of which RNA can be read to make proteins is time. Previous thinking was that a cell under threat would quickly make these RNAs in the nucleus and immediately push them into the cytoplasm.
College student betting on sports on phone while holding basketball
Health Lab

Sports gambling on the rise among college students

Betting on sports by college and university students, which has risen since most states legalized online gambling, raises potential addiction risks and needs more research and attention, experts say.
Blue and yellow illustration of DNA helix
Health Lab

New anti-cancer strategy shows promise in preclinical studies

New cancer therapeutic strategy rewires protein interactions to improve efficacy and reduce potential side effects.
Closeup of newborn baby, whose head is held by a blue medical gloved hand. Blue gloved hand also holds a stethoscope to baby's chest.
Health Lab

More babies with hypoplastic left heart syndrome surviving critical period between surgeries

Collaborative effort helped sustain improvements in survival for babies born with hypoplastic left heart syndrome (HLHS), a serious congenital heart condition that requires multiple surgeries.
Rachel Upjohn Building groundbreaking ceremony
News Release

How U-M helped bring depression out of the darkness

The history of the founding of the Eisenberg Family Depression Center at the University of Michigan, the nation's first center focused on research, community outreach and stigma reduction in depression and bipolar disorder.