U.S. surgeons use HoloLens 2 to remotely guide Brazilian doctors in delicate surgeries on baby brain models

CheckCitation/SourcePlease click:XR Navigation Network

(XR Navigation Network March 11, 2024) Benjamin Warf, a renowned neurosurgeon at Boston Children's Hospital, is at MIT's MIT.nano Immersion Lab. more than 3,000 miles away, his avatar is standing next to Matheus Vasconcelos in far-away Brazil and guiding the this resident perform delicate surgery on a doll-like model of a baby's brain.

Putting on a pair of AR/VR headsets, Vasconcelos can view Waugh's avatar demonstrating a brain surgery procedure before trying to replicate it himself and asking Waugh's digital twin questions at any time.

Worf said, "It's almost an out-of-body experience. Maybe that's what it's like to have an identical twin?"

That was the team's goal: to create immersive medical teaching across continents. Among other things, Waugh's digital twin can cross geographic boundaries and be effective around the world.

Vasconcelos, a neurosurgery resident at the São Paulo School of Medicine in Brazil, said, "This is the first time I've trained with the model, and it's performed very well. As a resident, having access to the professor's guidance, I now feel more confident and comfortable performing surgeries on real patients."

Waugh's avatar is a new project based on AR/VR startup EDUCSIM. In March 2023, EDUCSIM's Giselle Coelho began working with MIT technicians to create Waugh's avatar. By November of the same year, this avatar had begun training future surgeons like Vasconcelos.

Giselle said, "I had the idea of creating an incarnation of Dr. Walt as a proof of concept, and I was thinking, 'What part of the world would someone be working on technology like this?' . Then I found MIT.nano."

As a neurosurgery resident, Giselle was often so frustrated by the lack of practical training options for complex surgeries that she built her own model of a baby's brain. The physical model contains all the structures of the brain and can even bleed, so it is able to "simulate all the steps of surgery" from incision to skin closure.

Giselle soon found that simulators and AR/VR demonstrations reduced her learning curve, so she founded EDUCSIM in 2017, thereby expanding the variety and scope of training for residents and specialists looking to learn new technologies.

To create the avatars, Giselle worked with Talis Reks, an AR/VR/gaming/big data IT technologist at MIT.

One of the biggest challenges, Ricks explains, is helping to develop what Giselle calls "holographic transmission," which is the real-time transmission of Waugh's 3D video over the Internet, so that his avatar can be present at transcontinental medical trainings.

The solutions in question are synchronous and asynchronous modes. Vasconcelos is trained in asynchronous mode, in which you can observe a demonstration of the avatar and ask it questions. The system is able to work through artificial intelligence algorithms that draw on previous research and the large number of questions and answers provided by Waugh, and is available in multiple languages.

As for Sync Mode, Worf can operate his avatar in real time from a distance and make real holographic calls in real time.

In fact, Waugh has a long history of training much-needed pediatric neurosurgeons around the world. He notes that since the outbreak, distance and simulation training has become an increasingly large part of training, so "if one day we can have avatars like Giselle showing people in faraway places how to do things and answering questions for them, without any travel costs, without any time costs, etc., I think that would be a very powerful solution."

According to Giesel, such programs are especially important for surgeons serving in remote and underserved areas, such as the Brazilian Amazon: "In this way, they have access to the same level of education as elsewhere and the opportunity to get in touch with Dr. Wolff."

For example, a baby recently treated for hydrocephalus at an Amazon clinic had to travel 30 hours by boat to be able to undergo surgery. By training surgeons virtually, we "can change not only the reality of this child, but the future."