The Ultimate Guide to Augmented Reality in Surgery

While the physical world is 3D, most data is trapped on 2D pages and screens. This gulf between the real and digital worlds limits our ability to exploit the volumes of information available to us.

Augmented reality, a set of technologies that superimposes digital data and images on physical objects, is closing this gap.

What is Augmented Reality?

Augmented reality is a technology that superimposes computer-generated layers of digital information (graphics, images, or text), onto a view of the physical, real-world environment. It makes reality more interactive.

Augmented reality (AR) is part of extended reality (XR). XR refers to all technologies that combine the real and digital worlds to create a new kind of experience. It includes Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR).

What are the 2 main Types of Augmented Reality?

01. Marker-baser AR

Marker-based, or image recognition augmented reality, uses a trigger object as a cue to display content. The most common example of a trigger is a QR code. Marker-based AR can usually only be used with mobile devices like smartphones or tablets.

02. Markless AR

Markless AR is more versatile than marker-based AR. This type of AR uses cameras, GPS, and accelerometer information to track where the user is and display relevant information. This combination of inputs is known as Localization And mapping (SLAM).

What are the 4 Types of Markless Augmented Reality?

01. Location-based Augmented Reality

Location-based AR maps the real-world environment and defines visual positions in your surroundings. Once your device detects a match with the mapped location, it superimposes digital imagery accordingly.

Pokémon GO is the most well-known example of location-based AR.

02. Projection-based Augmented Reality

AR projects immersive light onto a flat surface to create 3D imagery. It then uses SLAM to detect human interaction with augmentation. For instance, projection-based AR can be used to create holograms, like functioning keyboards.

This type of AR needs a screen or headset and allows users to create surreal experiences for large audiences.

03. Contour-based Augmented Reality

Contour-based augmented reality, or outlining AR, leverages SLAM to outline the shapes of objects and simulate real human interaction.

For example, it might be used to develop a safe-driving application for nighttime and to guide pilots toward landing strips.

04. Overlay Augmented Reality

Overlay AR or Superimposed AR uses object recognition to partially or entirely replace an object within the user's environment with a digital image.

For example, a surgeon can use this type of AR to add a digital X-ray over part of a patient's body during an operation.

What does it take to use augmented reality?

Augmented reality needs a camera and AR software. This device can be a smartphone, tablet, smart headset, or glasses.

The user can control digital objects by touch screen, voice, or gesture.

AR needs specific components to create a truly realistic and immersive experience:

• Input devices: The input devices include cameras and sensors, which are used to capture the real-world environment, this data is then sent for processing.

• Computer processors: depending on the device, the processor or processors will be different. The data sent by the input devices are then processed using algorithms to track location, orientation, and objects in the real-world environment, allowing for accurate placement of digital objects. Objects are also rendered before being integrated into the physical environment.

• Output devices: also known as screens, projectors, or other displays, show the combined digital and physical world for a seamless augmented reality experience.

How does augmented reality work

Augmented reality effects are achieved by overlaying digital information onto a real-world environment via a headset or a smartphone application.

True AR experience also includes other sensory modalities as well, such as sound, touch, and olfactory, and typically requires special accessories to further enrich the environment.

Augmented reality superimposes 3D virtual content (image, text, video, graphics, information...) in the real world, and here in the surgeon’s field of vision.

AR uses sensors to understand the surrounding environment. Augmented reality uses a combination of GPS, accelerometers, and gyroscopes, to understand where a user is and what direction he is facing, and then overlay the artificial virtual content on the real world.

Augmented reality provides visual content and other sensory information to the surgeon through a device.

When a user points to the device and looks at an object, the software recognizes it with computer vision technology (CV). The device then downloads information about the object from the cloud. The information is then presented in a 3D form superimposed on the object.

The user sees the 3D digital object superimposed on the real object.

The most common AR devices are mobile phones. They have all the components necessary to make AR experiments possible, meanwhile, smart helmets or goggles give the most immersive experiences.

Despite its long history, AR has only recently made its debut in medical practice, being applied primarily to surgical navigation, taking data from preoperative imaging and using it in the surgical field to connect the two representations in real time.