IE 11 is not supported. For an optimal experience visit our site on another browser.

Augmented reality lets doctors peer inside the body like never before

New devices will end ‘historic disconnect’ in doctors's treatment of patients.
Image: Two dimensional data displayed on the torso of a medical mannequin
Two-dimensional data displayed on the torso of a medical mannequin.University of Alberta ProjectDR

Augmented reality (AR) technologies that blend computer-generated images and data from MRI and CT scans with real-world views are making it possible for doctors to “see under the skin” of their patients to visualize bones, muscles, and internal organs without having to cut open a body.

Experts say AR will transform medical care by improving precision during operations, reducing medical errors, and giving doctors and patients alike a better understanding of complex medical problems.

Augmented reality could help doctors determine exactly where to make injections and incisions. In medical emergencies, it could be used to display life-saving information for AR-equipped paramedics and other first responders: “It could have ‘push here’ [over the heart], and on the other side of the person's chest are the instructions for what to do,” says Dr. Greg Kawchuk, a professor of rehabilitation medicine at the University of Alberta in Edmonton, Canada, and a leader in medical applications of AR.

Kawchuk is a researcher on ProjectDR, an augmented reality system that can map internal medical scans into three-dimensional images overlaid on a patient’s body, either with a video projector or via AR smart glasses. He says AR will eliminate the “historic disconnect” between a doctor’s efforts to understand data from scans and other diagnostic tests and those to care for flesh-and-blood patients.

Augmented surgery

In a cutting-edge use of augmented reality in medicine, doctors at Imperial College and St. Mary’s Hospital in London have been wearing Microsoft’s HoloLens AR glasses during reconstructive surgery on patients who have suffered severe leg injuries in traffic accidents.

“You look at the leg and essentially see inside of it,” says Dr. Philip Pratt, a medical research fellow at Imperial College. “You see the bones, the course of the blood vessels, and can identify exactly where the targets are located.”

a. CTA imaging showing the location of perforating arteries with yellow arrows. b. An example HoloLens rendering of segmented polygonal models.European Radiology Experimental 2018

Doctors often repair severe leg injuries with flaps of tissue taken from elsewhere on the body. Connecting it to blood vessels at the site of the wound helps fresh oxygen-carrying blood reach the new tissue and keep it alive.

Surgeons have typically used a handheld scanner to locate the major blood vessels near the wound. But the augmented reality system helped surgeons find those blood vessels directly, by highlighting them in the three-dimensional virtual image displayed in an AR headset.

“Augmented reality offers a new way to find these blood vessels under the skin accurately and quickly, by overlaying scan images onto the patient during the operation,” Pratt says.

Ultrasound through AR

Dr. Amitabh Varshney, director of the University of Maryland’s Institute for Advanced Computer Studies, and Dr. Sarah Murthi, an associate professor of surgery at the university’s medical school, have developed an AR system that displays real-time ultrasound data directly onto a patient’s body through an AR headset.

The system lets doctors see data from an ultrasound probe displayed directly on a patient’s body, rather than on a computer screen — providing a single unified view instead of forcing doctors to divide their attention between the patient and a screen.

Varshney sees this system and AR in general as fundamentally altering medical care for patients and doctors alike — especially as AR moves beyond its roots in gaming and entertainment. “This is definitely not a game anymore,” he says. “These tools present a historic opportunity to transform medicine and patient care.”