Applying Extended Reality Technologies to Rounds

In an era of endless Zoom meetings and remote learning, virtual experiences can seem like a poor substitute for the real thing.

In some settings, however, a more immersive virtual experience may enhance or even improve upon the in-person one. Enter the Mixed Reality telerounding pilot that recently brought entire medical teams—and other institutions—to Michigan Medicine patients' bedsides virtually.

Mark Cohen, M.D., an endocrine surgeon at Michigan Medicine, realized while teaching via Zoom this past June that virtual reality and augmented reality have applications beyond entertainment. After reading about how the Imperial College of London was using the technology he connected with Jeremy Nelson, director of the XR Initiative for the Center for Academic Innovation at the University of Michigan, and learned other researchers were working on incorporating the technology into medical contexts. 

From curiosity to collaboration to creation

Nearly 50 faculty and leaders from several different departments on the medical campus soon formed a new collaborative and launched the IRB-approved Mixed Reality telerounding pilot, employing Microsoft's new HoloLens2 to enable solo rounding while bringing learners and colleagues into the room with the device wearer. 

In his vice chair role guiding telemedicine efforts in the Department of Surgery and the health system, Cohen saw opportunities to leverage the technology to advance how patients are seen remotely, and to better engage residents and learners at Michigan Medicine's many satellite hospitals, surgical centers and clinics. 

"The more we delved into it, the more ideas and use cases we were coming up with" Cohen says." 

The HoloLens2 brings people into the room with the wearer via a high-definition camera and allows the operator to pull up images and files via a heads-up display for others to see.

Folding the technology into the rounding experience was logical with COVID-19 disrupting in-person experiences; fewer students and team members in a room lessened the exposure risk to them (and to the patient), and also conserved precious PPE. One person with a Hololens2 donning gowns and gloves to see each patient uses 600% less PPE use than an average in-person rounding team, according to Cohen.

For the pilot, mobile carts were outfitted with the Hololens2, laptops on a HIPAA-compliant secure network, 3D-printed face shields for use under the headset and a Cleanbox to sterilize the equipment between users. After trialing this telerounding with several faculty, and bringing in remote experts from the Imperial College of London, the group hosted 155 attendees for a Mixed Reality International Grand Rounds via Zoom in December.  

Faculty, residents, medical students, administrators, health system and medical school leadership and a cohort from the Imperial College of London saw examples of rounding and teaching on the wards and in the OR, as well as pulling in imaging and holographic teaching tools into the environment from the patient's bedside. Attendees could interact with the patient as if they were in the room. 

Virtual can be richer, and maybe even better

Bringing the HoloLens into patient rooms was prompted by a need to minimize: Fewer people, fewer potentially risky interactions. It was soon clear to Cohen that the experience broke down several virtual learning barriers, and addressed some of the disparities in learning for medical students who were unable to examine patients in person due to the pandemic.

Now students who had to remain virtual or practice social distancing could be brought right to the bedside and interact directly with patients again.

"Students thought the audio-visual quality was remarkable. One of the students actually told me that they saw more through this process than they would standing in the back of the room in a big rounding group in person," Cohen says. 

A recent survey of attendees from the Mixed Reality International Grand Rounds found that 97% felt the ability to visualize the patient's clinical findings and imaging in a virtual rounding experience is valuable, while 98% felt there was a benefit to the multi-center collaborative experience in an international XR grand rounds format over a traditional grand rounds format.

MORE FROM THE LAB: Subscribe to our weekly newsletter

The ability to quickly bring up images and other materials for viewers on the fly also made for a richer experience; where a physician would not normally have easy access to all of a patient's files on mobile rounds—or the ability to easily display them—Cohen could easily bring up holographic windows to show images and labs for those along for the virtual ride. 

The feature also meant a physician could go deeper into actual teaching at the bedside, facilitating a conversation about the disease process. 

"We realized that having this ability to bring people to the bedside more easily meant you could have a better interaction virtually with the patient and show some really cool physical exam findings, bring in holographic windows and show imaging and labs and enhance the educational experience of rounds," Cohen says.

Although new and different, patients adapted

Perhaps Zoom and other virtual technologies prepped patients for a world where a surgeon would appear bedside with futuristic goggles on.

Even though patients were told ahead of time he'd be wearing the headset and that others would be present virtually for the visit, Cohen was still struck by how relatively normal the interactions were, and how open patients were. 

In fact, patients seemed to forget anyone else was along for the visit until a voice chimed in from the headset or the laptop on the mobile XR cart Cohen used to transport the equipment.

"One of the things that was really interesting was when one of the students asked a question about a patient's postoperative course. You could see that it then kind of dawned on the patient that there was this whole group listening in and that it wasn't just me," Cohen says.

Just as students might benefit from not being relegated to the back of a rounding group, patients may be more at ease without a crowd in front of them.

"People are more at ease talking to one person about their problems then in front of a whole group. This breaks down some communication barriers with patients we see during traditional team-based in-person rounds. Many patients become easily intimidated when an entourage of people in white coats and scrubs walk into their room and surround their bed. Having just one person to talk to, even with a headset on, allows for a more comfortable connection and discussion," Cohen says. 

If anything seemed markedly different, it was a result of the physical headset itself. Cohen found that patients sometimes tended to look at the sometimes-flashing light by the camera on the headset instead of his eyes. But this quickly became less important as they talked about their disease and symptoms. 

Potential beyond rounds

Like many things, the COVID pandemic catalyzes the adoption of new technologies like telemedicine, Zoom, and now XR-based tools for the clinic. 

Aside from enhancing how clinicians round and teach on the wards and ICUs while lowering exposure and PPE usage, Cohen sees a much greater potential for XR technologies in clinical care delivery and surgical training. One example is advancing real time virtual collaboration in surgery. 

"Imagine you are a surgeon in the UP [Upper Peninsula of Michigan] or even in a battle field on the other side of the world and are performing a liver resection, and you get stuck in a tough area of the dissection. With an XR headset, you can remote into your headset an expert surgeon like Dr. Cliff Cho to help you walk through a tough spot in the case and help you save a life," Cohen says. 

Cohen also sees applications for collaboration working the other way:

"Imagine using a tool like this to remotely proctor surgeons from around the state or around the country on their operative techniques, or to certify them to perform new procedures, or for sharing best practices to improve and better-standardize quality across a larger network. The possibilities are astounding when you think about all the use-cases XR technology can be applied to in surgery," Cohen says.  

Like Podcasts? Add the Michigan Medicine News Break on iTunes, Google Podcast or anywhere you listen to podcasts.

Such support could especially benefit surgeons in more far-flung areas, where they might not see certain cases as frequently.

Resources vary among hospital systems, of course, but Cohen is optimistic that this particular setup is relatively accessible. In fact, the all-in cost for a complete mobile XR cart including the Hololens2 headset with native software, a laptop, cleaning supplies, and the cart total less than $5,000. 

The team will continue to conduct patient encounters with the HoloLens2 and collect rounding data over the next several months from at least 100 patients, analyze the data and see where it leads them with broader utilization. 

"The more we're able to study it the more easily we'll be able to integrate that into how we deliver clinical care," Cohen says.

There are currently three mobile carts available for use in the Department, and faculty or residents interested in learning how to use the technology or try it for their rounding may contact Dr. Cohen.

No matter what the exact implementation looks like, there's no going back.

"One of the exciting things about this whole process is we are breaking down barriers with this technology. Whether it is access barriers for patients, learning barriers for our medical students, or connectivity barriers to share expertise and best practices with our colleagues around the world, XR applications are not just a reaction to a pandemic, but a function of our future environment. It has potential to dramatically enhance how we teach, how we practice medicine and how telemedicine on remote connectivity can take on a more immersive dimension," Cohen says.

It takes a village, and creative solutions to barriers

If dozens of people conceived the pilot, just as many helped see it through. The project presented many barriers, and a diverse and supportive team was crucial to overcoming them.

Students and residents stepped up to work out how to connect the headsets, share images and integrate the technology into the learning environment and on virtual calls. 

"Significant praise and appreciation should go out to all the students and residents like Taylor Kantor, Arushi Mahajan, Donovan Inniss and Michelle Benedict who spent many hours helping to work this out," Cohen says. 

Beyond technical issues, logistic and HIPAA compliance issues needed navigating; multiple systems needed to talk to one another, and anything dealing with patient information needed to be secure. Cohen credits Microsoft team members, Jeremy Nelson and the CAI XR initiative, collaborators at Imperial College, leadership at Michigan Medicine and the Medical School and the HITS team and its leadership for making the pilot and International Grand Rounds possible.  

"There were many barriers to get to this point that had to be broken down," Cohen says.