Prediction models, risk scores and decision tools are becoming a more integral part of modern medical practice. Current systems have demonstrated that early detection of patient deterioration can lead to reduced mortality risk, reduced length-of-stay, and decreased hospital costs. But to implement these systems seamlessly within hospitals outside of where they were originally developed, validation at multiple institutions with significantly different patient populations is critical.
In a recent study published in Critical Care Medicine, researchers from Michigan Medicine, the Weil Institute for Critical Care Research & Innovation, Precision Health and Hurley Medical Center in Flint, MI analyzed data from more than 11,000 hospital encounters to externally validate a predictive analytic tool called PICTURE, or Predicting Intensive Care Transfers and other Unforeseen Events. PICTURE is a machine learning algorithm that utilizes electronic health record data to predict intensive care unit transfer or death as a proxy for patient deterioration in hospital settings.
In the study, the team examined whether the predictions made by PICTURE, which was developed at U-M Health, would generalize to a second community-oriented hospital that had significantly different patient demographics.
According to the study, moving a predictive model to a new clinical environment outside that which it was trained is fraught with challenges. Every hospital is unique and will have elements that can’t be controlled for or replicated, note the researchers. Just because the predictive model showed excellent performance at the institution where it was originally trained doesn’t mean that same model will perform equally well in a different environment.
“In the paper, we wanted to investigate the similarities and differences between Hurley Medical Center and Michigan Medicine, and show how PICTURE responds,” said Brandon Cummings, a senior data scientist at the Weil Institute and lead author of the study.
Michigan Medicine is a large academic research institution and medical center where many patients are referred for advanced specialty care. In contrast, Hurley Medical Center is a large 443-bed community hospital providing medical services to Flint, MI and Genessee county.
Although both hospitals use the same EHR vendor (Epic Systems), differences in their patient populations, clinical care guidelines, demographic makeup, and many other factors led to changes in feature distribution, deterioration rate, and data completeness that had the potential to impact the model.
“A common hurdle in developing a tool that can predict clinical deterioration is showing that the tool can be used across multiple institutions,” said Mike Roebuck, M.D., Chief Medical Informatics Officer and an emergency physician at Hurley Medical Center, clinical assistant professor of emergency medicine at U-M, and co-author of the study. “PICTURE performed very well at both institutions at predicting meaningful clinical deterioration.”
Despite the clinical and information-technology differences, PICTURE was able to consistently predict deterioration events and outperform existing metrics at both institutions, including a well-known proprietary system called the Epic Deterioration Index, supporting its suitability as an early-warning reminder tool to predict deterioration in general ward patients across different clinical settings.
The research team said they are encouraged by the results and are looking forward to validation across further hospitals to ensure portability across a wider variety of clinical settings.
At University of Michigan, the Weil Institute is rolling out PICTURE to the Rapid Response Team – a group of nurses and other health care professionals who bring early intervention critical care to the bedside of patients who demonstrate acute changes and/or are progressively deteriorating. They are also working closely with Precision Health at U-M to carry out clinical studies on additional iterations of PICTURE predictive models.
“Conducting high-quality studies of predictive models is difficult and requires coordination across many different stakeholders and tackling of technical barriers. Precision Health is excited to collaborate and provide resources to help teams like the Weil Institute answer important questions around the implementation of predictive models,” said Karandeep Singh, M.D., assistant professor of learning health sciences in the Department of Internal Medicine and Division of Urology at the U-M Medical School, assistant professor of information at U-M and associate chief medical information officer for Artificial Intelligence.
Paper cited: “External Validation and Comparison of a General Ward Deterioration Index Between Diversely Different Health Systems,” Critical Care Medicine. DOI: 10.1097/CCM.0000000000005837
Disclosures: Cummings’s, Blackmer’s, Farzaneh’s, Cao’s, and Ansari’s institutions received funding from the Michigan Institute for Data Science and Airstrip Technologies. Cummings, Blackmer, Farzaneh, Cao, and Gillies, Medlin, Ward, Sjoding, and Ansari have disclosed that multiple patents have been filed for this work. Their invention disclosures have been submitted with the Office of Technology Transfer, University of Michigan, Ann Arbor and Airstrip Technologies has a license option for Predicting Intensive Care Transfers and other UnfoReseen Events from the University of Michigan. Motyka disclosed that he is currently employed by at Strata Oncology (Precision Oncology). Gillies disclosed that he is employed at Regeneron Pharmaceuticals. Admon’s institution received funding from the National Heart, Lung, and Blood Institute. Admon and Sjoding received support for article research from the National Institutes of Health. Singh’s institution received funding from Blue Cross Blue Shield of Michigan and Teva Pharmaceuticals; he received funding from Flatiron Health. Sjoding’s institution received funding from the NIH. The remaining authors have disclosed that they do not have any potential conflicts of interest.