Study of 24 U.S. hospitals shows onset of COVID-19 led to spike in sepsis alerts

The study highlights the importance of AI oversight to prevent alert fatigue, and the effects of “dataset shift” on widely implemented models.

12:18 PM

Author | Clarissa Piatek

red air horn white room
Getty Images

A descriptive study in JAMA Network Open—involving 24 hospitals across four geographically diverse health systems (the University of Michigan, New York University Langone Health, Mass General Brigham, and BJC Healthcare)—looked at the number of sepsis alerts in the months before and during COVID-19—and found a significant jump in alerts after the start of the pandemic, despite lower patient numbers from canceled elective surgeries.

The Epic Sepsis Model, also known as ESM, is a widely implemented artificial intelligence based system that will generate an alert if a patient displays a sufficient number of variables indicating a risk of sepsis.

"The ESM calculates a score from 0-100, which reflects the probability (or percent chance) of a patient developing sepsis in the next 6 hours," explained author Karandeep Singh, MD, MMSc, an assistant professor in the Departments of  Learning Health Sciences, Internal Medicine, Urology, and Information at U-M. Singh also heads U-M Health's Clinical Intelligence Committee.

"We use an alerting threshold of 6 at Michigan Medicine on the inpatient side, and many other institutions use a similar threshold of 5 or 6 to generate alerts."

With an alerting threshold of 6, "our study found that the total volume of daily sepsis alerts increased by 43%, even though hospitals had cancelled elective surgeries and reduced their census by 35% to prepare for the [COVID] surge," Singh said. One reason for the higher number of alerts despite a lower patient census is that the patients who filled hospitals during COVID's first wave were, in general, in more critical condition than elective surgery patients and therefore more likely to generate alerts.

Another reason for the jump in alerts could be a result of "dataset shift," where a model's performance will deteriorate when there are sudden, unexpected changes in a hospital's case mix (e.g., a COVID surge). Singh previously described the phenomenon of dataset shift in a July 2021 New England Journal of Medicine paper.

"COVID-19 was a 'black swan' event that likely affected many existing models in ways we don't yet fully understand," said Singh. "Future studies that use post-pandemic data to evaluate AI models should be careful to interpret their findings within the context of the pandemic."

And aside from what caused it, "the increase in total alerts is illuminating because it provides a sense of just how busy front-line clinicians were during the first wave [of COVID], despite the lower hospital census," he said.

During the first wave of COVID, "at Michigan Medicine, nurses qualitatively reported a rise in sepsis model-driven alerts at levels that felt untenable," he continued. "Although our study does not quantify alert fatigue, our primary motivation for looking at sepsis scores during this period was that nurses at Michigan Medicine reported a large rise in alerts that ultimately led to these alerts being paused": a pause that corresponded to an increase in alert volume. Singh explains that the "rise in sepsis scores appears to have been a common finding at many hospitals around the country."

The study points to the importance of monitoring sudden increases in alert volume, both to prevent alert fatigue for overworked healthcare workers, and to look into whether some form of dataset shift is affecting a model's accuracy.

"We need to have a way to anticipate and deal with situations where the alert volume is high," said Singh. "Even among well-resourced hospitals, there is a limit to the capacity to respond to model-driven alerts. Clinical AI governance is a first step toward providing real-time guidance to health systems when situations like this arise, and for setting standards based on local constraints."

Paper cited: "Quantification of Sepsis Model Alerts in 24 US Hospitals Before and During the COVID-19 Pandemic," JAMA Network Open. DOI: 10.1001/jamanetworkopen.2021.35286

More Articles About: Rounds Covid-19 Health Care Quality medical devices infectious disease
Health Lab word mark overlaying blue cells
Health Lab

Explore a variety of healthcare news & stories by visiting the Health Lab home page for more articles.

Media Contact Public Relations

Department of Communication at Michigan Medicine

[email protected]


Stay Informed

Want top health & research news weekly? Sign up for Health Lab’s newsletters today!

Featured News & Stories sketched out bacteria in a dish yellow and blue colors of U-M
Health Lab
This gross mixture has big benefits for the study of bacteria
Michigan Medicine researchers have found that growing bacteria on agar mixed with organs is an efficient and effective way to study infectious pathogens.
patient giving paperwork and person saying no with hand graphic moving teal white grey navy orange
Health Lab
Why new patient paperwork isn’t just busy work
While it’s easy to overlook doctor's office questionnaires, that paperwork actually serves a vital role in better understanding how to treat you. Called patient reported outcomes, this information gives medical specialists insight into how treatments truly impact you as a patient.
supar molecule teal blue yellow red
Health Lab
Immune protein suPAR links viral infection as possible cause of kidney disease
Through a series of experiments in non-human primates, mice and humans, a multi-institutional team led by researchers from Michigan Medicine and Rush University found that the immune protein soluble urokinase plasminogen activator receptor, or suPAR, is an important link between viral infections and proteinuria; the elevation of protein in the urine is known to cause glomerulopathy, a common form of kidney disease.
person holding walker with nurse next to them closer up on hands lower body
Health Lab
Long COVID happens in nursing homes, too
Post-acute sequelae of Sars-COV2 (PASC, long COVID) caused a decrease in independence and cognitive ability after coronavirus infection in nursing home residents
wheelchair walker image
Health Lab
Spread of drug resistant bacteria linked to patient hand contamination and antibiotic use within nursing homes
A Michigan Medicine research team seeks to identify characteristics of patients within nursing homes, as well as the nursing home environment itself, that are associated with contamination by vancomycin-resistant enterococci.
daycare kids running around in room
Health Lab
Child care centers aren’t a likely source of COVID-19 spread, study says
Research published from experts at Michigan Medicine, the University of Pittsburgh School of Medicine, and UPMC Children’s Hospital of Pittsburgh shows that children in daycare centers were not significant spreaders of COVID-19