Beam me up, doc: Adding touch and feel to virtual reality

Imagine sitting down in your doctor’s office, putting on a headset and being transported to an examination room. A specialist sits in front of you and, after a short conversation, touches your stomach to locate the source of persistent pain. You feel like you’re in the same room, but in reality, you’re hundreds of kilometers from each other.

On the heels of his successful demonstration last summer of holographic technology, Adam Sirek, a family physician and faculty member at both Western University’s Schulich School of Medicine and Dentistry and at Western Space, now is working on adding touch and feel to the technology.

During his demonstration, Sirek’s team at Western was able to conduct a physical examination on a patient’s shoulder in Alabama through the hologram’s 3D motion.

“It’s hard to describe the difference the holographic technology makes, but you get that internal sense that the person is actually there, and you have a much more meaningful interaction by seeing that 3D form in front of you,” says Sirek.

Rather than simply logging on to a video-conferencing platform to talk to a 2D image of the person on the other end, holoportation, created by Microsoft Research, creates a 3D image that allows the users to interact as though they are in the room together.

Sirek’s background as a family physician guides his interest in high-tech solutions. He says he isn’t thinking about how this technology could be used by a cardiothoracic surgeon but rather, “Is this going to be relevant immediately to me and my patients?”

A recent study titled Why they leave: Small town rural realities of northern physician turnover found that burnout and lack of amenities influence rural physicians’ decisions to leave their practices in Northern Ontario.

Mindy Smith, a family physician, professor at Michigan State University and former president of the Kooteney Boundary Patient Advisory Committee and Community, knows this challenge well, both as a physician and as a patient. Smith had to travel four hours from her home in rural southeastern British Columbia to see a specialist. In the end, her appointment only lasted five minutes.

Smith says the benefit of telemedicine in rural communities, both for clinicians and patients, is increasing the availability of specialists who aren’t present there, saving patients the time and expense of travelling long distances for short, simple appointments.

The holoportation system is made up of two devices. One is a Hololens, a head-mounted display that looks and operates like a virtual-reality headset.

“It basically creates a three-dimensional screen in front of your face so you can see the hologram,” explains Sirek.

The other device is a camera that scans the user to create the hologram. The camera utilizes lidar scanning technology using lasers to measure distance and depth to create a high-quality 3D image. The camera used in Sirek’s July 2022 demo is a Microsoft Kinect camera, but Sirek says that the holoportation software has since become device agnostic, meaning less expensive camera models can be used. Cameras that use lidar scanning technology are common – similar cameras come in Xbox gaming consoles as well as in the latest smartphones and tablets.

Originally, this technology lived in the realm of Sirek’s other passion – aerospace medicine. Holoportation opened possibilities to provide health care to astronauts while they are in low Earth orbit, where they may not have access to a doctor, let alone a specialist. Holoportation also provided opportunities for astronauts to connect with family members in a meaningful way during extended stays in space.

However, Sirek admits that while the holoportation technology greatly improves upon video conferencing by adding the 3D element, an essential component is still missing.

“The biggest failing of all of the virtual technologies right now is that I can’t do a physical exam.”

“The biggest failing of all of the virtual technologies right now is that I can’t do a physical exam,” explains Sirek. “I could talk you through lifting your arm, moving it and things like that, but I can’t feel it. I can’t feel grinding, I can’t feel catching, I can’t do the things that I could do in the office with you.”

Sirek and the medical students on his team at Western University are trying to address this by incorporating biosensors. Sirek explains that a third device embedded with biosensors could be placed on the patient while the physician on the other end activates those sensors remotely. The sensors would transmit force feedback to the physician, allowing them to feel the patient’s body.

“That sort of tactile feedback is the next step that we’re looking at from the medical standpoint because that will hugely improve the capabilities of the technology and then give us that physical examination that is so key to a lot of what we do in diagnosing things in-person,” says Sirek.

This is another area where Sirek’s two passions overlap: pilots of big planes like the Airbuses used for transatlantic flights use “fly-by-wire” control systems that give the pilot the sensation of what flight controls are doing through electronic input. The sensations are transmitted from the plane’s wings to the cockpit via electronic signals. Sirek says he hopes to take that same principle and extend the distance that the sensation is travelling.

In its most simple form, the technology that Sirek is looking to replicate is, once again, likely already in your house – it’s what makes a video game controller shake in response to what’s happening in the game.

Though he doesn’t think holoportation will ever replace an in-person exam, Sirek says he is hopeful that the inclusion of touch sensors is a further step in bringing patients and their care providers closer together even when they’re hundreds of kilometres apart.

Smith says that despite the surge in use of telemedicine and virtual care during the pandemic, her research has shown that the majority of patient visits using telemedicine technology during the pandemic were over the phone.

“Clinicians worry a great deal about missing clinical signs when they’re on the telephone and even on video, and there aren’t really clear guidelines for when it’s appropriate,” says Smith.

Though Smith says the data doesn’t support clinicians’ assumption, the concern persists. Smith says it’s a lot like starting patient visits by listening to a patient’s heart and lungs, even if the issue they’re seeing a physician for doesn’t involve those organs.

“I think wanting to see patients in the office is the same thing. It’s a long tradition, it’s a habit. It’s where the comfort zone is,” says Smith. “And so, people are reluctant to use the digital technologies that are available that they’re using every day with their family.”

Sirek says he hopes that holoportation and the potential for biosensors will help alleviate some of these concerns. If physicians can conduct virtual visits in which they feel like they’re in the room with patients and can touch them to assess them, it’s possible that they will be more likely to utilize virtual technologies.

Even once holoportation is fully developed, a significant hurdle to implementing it in rural and remote communities is Internet access. Initiatives like B.C.’s Digital Health Strategy that provides funding for high-speed internet infrastructure in 200 rural and Indigenous communities and SpaceX’s satellite internet service Starlink are broadening Internet access across the country. For now, less than 50 per cent of Canadians living in rural and remote communities have reliable access to high-speed Internet.

Despite the hurdles, Smith says she hopes that new technologies will strengthen the relationship between patients and their care providers.

“What keeps both patients satisfied and clinicians satisfied with their care is their relationships, and technology has to enhance that, not disrupt that,” says Smith.