16. Basic Augmented Reality Demo (BARD)¶
This series of tutorials aims to teach some of the concepts used in medical Augmented Reality (AR) applications.
16.2. The Idea¶
Augmented Reality (AR) is where additional information is used to augment the visual display of a scene. In medical terms, this normally occurs in two guises:
Video see through: The “reality” is represented by a video image from a medical
device such as a laparoscope or endoscope, which naturally displays on a monitor.
The overlaid information could come from pre-operative MR/CT data as shown above, or it could also be another live image modality. Take a look at this example from Xin Kang … Raj Shekhar and colleagues, from this article, where an ultrasound image is overlaid on the video. In this ultrasound example, the laparoscope and laparoscopic ultrasound are optically tracked.
In another example, Pratt et al. provide a pure vision based system:
In the vision system, the video camera provides the means for sampling the real scene, working out the pose of the ultrasound probe relative to the video camera, and the AR overlay itself.
Optical see through: The “reality” is directly observed via a visor, or semi-silvered mirror, onto which is projected additional information.
The most obvious example right now, is the Microsoft Hololens. See this example from ApoQlar, on the ApoQlar YouTube channel.
While we can’t develop such technology in a week, we can explore the general idea!
16.3. The Original Lab-Based Demo¶
We also have a CT scan of the phantom. The general idea is to take a webcam, register the CT coordinates to a reference marker, track the marker with a webcam and overlay the CT scan on the live video.
This demonstrates Augmented (adding the CT data), Reality (represented by webcam video) as an example of video see through AR.
16.4. The New Home-Based Tutorials¶
However, in 2020, due to the pandemic, we developed a new series of tutorials with the aim of exploring each component of an AR system in turn.
As we can’t actually do an overlay onto a physical object, due to everyone being at home, the overlay tutorial focuses on implementing something similar to Pratt et al, 2015 where the video camera itself is doing the tracking.
Each tutorial should be fairly stand-alone, and suitable for completing at home, or for distance learning.
These tutorials are: