The AI is listening: Implications of machine generated clinical notes on health care

Artificial intelligence tools are being touted as a solution for so-called pajama time – the time doctors inevitably spend compiling patient notes after working hours. 

As hospitals increasingly deploy AI scribes to listen and take notes during patient encounters, the data the scribes produce becomes the scaffold on which insights about health care, from billing to research to clinical decision making, is built. 

This rapidly changing infrastructure is the basis of a recent article written by Taofeeq Oluwatosin Togunwa, M.D. and Jodyn Platt, Ph.D., of the Department of Learning Health Sciences at University of Michigan Medical School. 

“It used to be that the doctor or clinician taking the notes was the filter for data. Now a machine is in the pipeline deciding what does and does not become data,” said Platt. 

Often, these systems are proprietary with opaque training data. 

And, what happens if the AI is not trained on a patient’s demographic, wondered Togunwa.  

“Usually, these models are trained on English and American accents and then they perform well within that distribution. But you find concerning literature where these models actually don't perform as well with people that might have different types of accents,” he noted.  

The ensuing hallucinations – fabrications introduced by AI when inputs do not match training data – then end up as part of the official dataset. 

And while clinicians are supposed to review machine-scribed notes for accuracy, “there isn’t a lot of incentive to go line by line if the note looks generally fine,” he said.  

Additionally, AI scribes can become less accurate over time if they are not routinely updated.  

“If real life data is substantially different from the training data, you start to have these challenges where there’s a drift in performance of the model. So, there’s a need to continuously audit these systems.” 

Beyond drift monitoring, to protect data infrastructure in the face of new technologies, Platt points to measurement. 

“Measurement is really important. We need to measure more things more often,” she said. “And not the same old things and not just the things that are easy to measure, but the things that we know are important.” 

And while reduced notetaking time is important, Platt says health systems should make sure benefits go both ways. 

“Are our patients also receiving that same level of benefit in their health care? For example, are they able to get to a diagnosis faster, or to see a clinician faster and are those benefits accrued equitably?” 

Paper cited: “Ambient AI Scribes as Emerging Infrastructure in the Learning Health System,” Learning Health Systems, DOI:  10.1002/lrh2.70115 

Sign up for Health Lab newsletters today. Get medical tips from top experts and learn about new scientific discoveries every week. 

Sign up for the Health Lab Podcast. Add us wherever you listen to your favorite shows.