Updated: Jun 28
1. OssoVR is a virtual surgery simulation platform that offers realistic hand-based interactions.
Source: OssoVR
2. One Shot Immersive develops virtual reality training experiences for mass casualty and other trauma emergencies. The products can be used to train medical and nonmedical personnel and enable them to make life-saving decisions in critical situations.
Source: One Shot Immersive
3. Surgical Theater created a VR-based surgical rehearsal platform for neurosurgical preoperative planning and rehearsal.
Source: Surgical Theater
4. Touch Surgery AI-powered surgical video and analytics platform for the OR, connect your OR to the cloud, so you can seamlessly record and upload your video. Video is automatically segmented into key procedural steps using their AI algorithms.
Source: Smallfry
5. Immersive Touch creates virtual/ mixed reality solutions for surgical planning, training, and education in neurosurgery.
Source: Immersive Touch
6. Vicarious Surgical’s robotic system, controlled by a VR headset, is designed to allow freedom of movement and precision in minimally invasive surgeries.
Source: Vicarious Surgical
7. Mentice Virtual Reality simulators for image-guided interventional therapies.
Source: Mentice
8. FundamentalVR delivers haptic simulators that let surgeons practice and sharpen their skills. Using realistic vibration patterns, the haptic technology gives doctors the feeling of holding actual tools.
Source: FundamentalVR
9. ZSpace creates hardware and software solutions to support the use of VR and AR for various industries. Its health science applications let users explore 3D human biology models, surgical and equipment safety procedures as well as over 4,600 anatomical structures in a VR environment.
Source: Zspace
Updated: Jun 28
While the physical world is 3D, most data is trapped on 2D pages and screens. This gulf between the real and digital worlds limits our ability to exploit the volumes of information available to us.
Augmented reality, a set of technologies that superimposes digital data and images on physical objects, is closing this gap.
What is Augmented Reality?
Augmented reality is a technology that superimposes computer-generated layers of digital information (graphics, images, or text), onto a view of the physical, real-world environment. It makes reality more interactive.
Augmented reality (AR) is part of extended reality (XR). XR refers to all technologies that combine the real and digital worlds to create a new kind of experience. It includes Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR).
What are the 2 main Types of Augmented Reality?
01. Marker-baser AR
Marker-based, or image recognition augmented reality, uses a trigger object as a cue to display content. The most common example of a trigger is a QR code. Marker-based AR can usually only be used with mobile devices like smartphones or tablets.
02. Markless AR
Markless AR is more versatile than marker-based AR. This type of AR uses cameras, GPS, and accelerometer information to track where the user is and display relevant information. This combination of inputs is known as Localization And mapping (SLAM).
What are the 4 Types of Markless Augmented Reality?
01. Location-based Augmented Reality
Location-based AR maps the real-world environment and defines visual positions in your surroundings. Once your device detects a match with the mapped location, it superimposes digital imagery accordingly.
Pokémon GO is the most well-known example of location-based AR.
02. Projection-based Augmented Reality
AR projects immersive light onto a flat surface to create 3D imagery. It then uses SLAM to detect human interaction with augmentation. For instance, projection-based AR can be used to create holograms, like functioning keyboards.
This type of AR needs a screen or headset and allows users to create surreal experiences for large audiences.
03. Contour-based Augmented Reality
Contour-based augmented reality, or outlining AR, leverages SLAM to outline the shapes of objects and simulate real human interaction.
For example, it might be used to develop a safe-driving application for nighttime and to guide pilots toward landing strips.
04. Overlay Augmented Reality
Overlay AR or Superimposed AR uses object recognition to partially or entirely replace an object within the user's environment with a digital image.
For example, a surgeon can use this type of AR to add a digital X-ray over part of a patient's body during an operation.
What does it take to use augmented reality?
Augmented reality needs a camera and AR software. This device can be a smartphone, tablet, smart headset, or glasses.
The user can control digital objects by touch screen, voice, or gesture.
AR needs specific components to create a truly realistic and immersive experience:
Input devices: The input devices include cameras and sensors, which are used to capture the real-world environment, this data is then sent for processing.
Computer processors: depending on the device, the processor or processors will be different. The data sent by the input devices are then processed using algorithms to track location, orientation, and objects in the real-world environment, allowing for accurate placement of digital objects. Objects are also rendered before being integrated into the physical environment.
Output devices: also known as screens, projectors, or other displays, show the combined digital and physical world for a seamless augmented reality experience.
How does augmented reality work
Augmented reality effects are achieved by overlaying digital information onto a real-world environment via a headset or a smartphone application.
True AR experience also includes other sensory modalities as well, such as sound, touch, and olfactory, and typically requires special accessories to further enrich the environment.
Augmented reality superimposes 3D virtual content (image, text, video, graphics, information...) in the real world, and here in the surgeon’s field of vision.
AR uses sensors to understand the surrounding environment. Augmented reality uses a combination of GPS, accelerometers, and gyroscopes, to understand where a user is and what direction he is facing, and then overlay the artificial virtual content on the real world.
Augmented reality provides visual content and other sensory information to the surgeon through a device.
When a user points to the device and looks at an object, the software recognizes it with computer vision technology (CV). The device then downloads information about the object from the cloud. The information is then presented in a 3D form superimposed on the object.
The user sees the 3D digital object superimposed on the real object.
The most common AR devices are mobile phones. They have all the components necessary to make AR experiments possible, meanwhile, smart helmets or goggles give the most immersive experiences.
Despite its long history, AR has only recently made its debut in medical practice, being applied primarily to surgical navigation, taking data from preoperative imaging and using it in the surgical field to connect the two representations in real time.
Updated: Jun 28
3D visualization of medical organ images is becoming more important in surgery. Understanding the patient-specific 3D anatomy helps surgeons better prepare for surgery, providing better results for patients.
Currently, conventional volumetric images such as MRI and CT or even ultrasound images are used to schedule surgery.
Mixed reality, which is the fusion of real and virtual worlds where physical and digital objects coexist and interact in real time, has already managed to provide holographic 3D images to surgeons.
What is mixed reality?
Mixed reality is the merging of real and virtual worlds where physical and digital objects co-exist and interact in real-time.
Mixed reality is the possibility to do both, virtual reality and augmented reality in the same app or on the same device.
The user views the real world while manipulating the digital content generated by the device.
Applications of mixed reality in surgery:
The 5 applications of mixed reality in surgery are:
Preoperative planning
Documentation of the intervention
Holographic 3D visualization of any imaging of the patient during surgery with the possibility to interact with it: immersion and interaction with the two real and virtual worlds
Collaboration of surgeons in real-time and collaboration of remote surgeons
Education of medical students and training of surgical residents
1. Preoperative planning
Mixed reality can be really useful for planning surgery. A surgical plan performed on a patient’s X-ray may be used during the operation and even after the operation for resident training in surgery or the patient.
Mixed reality has already been successfully applied as a planning tool in various surgical scenarios, including urology, thoracic surgery, neurosurgery, colorectal surgery, and bariatric surgery...
Mixed reality glasses such as HoloLens 2, and Holoeyes, facilitate the spatial localization of anatomical structures and improve mental alignment, which simplifies preoperative planning.
These smart glasses can also improve patient comfort and understanding with Virtual Surgery Intelligence for HoloLens 2.
2. Documentation of the operation
With mixed reality, any surgeon can document the procedure and then use images, videos, or audio to train residents or students.
3. Holographic 3D visualization during the operation
Mixed reality allows the process of reconstructive surgery. With reconstructive surgery, surgeons can locate and reconnect major blood vessels.
Holographic overlays help surgeons see the bones and identify the evolution of the blood vessels that help them with their surgery, improving the outcome for the patient.
Mixed reality helps healthcare professionals access 3D medical information and anatomical images of the patient during surgery.
The use of 3D models to estimate size and shape before performing surgery has been effectively implemented for almost a decade.
Mixed reality allows you to generate customized 3D models for each patient and visualize the internal anatomy in a fully immersive environment.
This opens up new possibilities, such as preoperative simulations, to determine optimal procedures and predict final surgical outcomes, it is a good method for any surgeon to refine his gestures and simulate with the surgical team the virtual operation before the day of the operation.
4. Collaboration between surgeons
Mixed reality also helps build a bridge between surgeons who work in different parts of the world to collaborate or even surgeons in the same operating room or hospital.
5. Education and training of students and residents
Mixed reality surgical simulators are becoming an important part of the training process for medical students and residents, as they allow an appropriate training environment to recreate realistic and reproducible scenarios without putting the patient at risk.
Mixed reality has already been evaluated as an aid tool during endoscopic procedures, spinal surgery, interventional radiology procedures, and orthopedic surgery.
Mixed reality allows a better understanding of the specific anatomy of each patient and thus improves surgical planning for more precise, safer, and less invasive surgical operations.
Mixed reality can include complex information such as holographic images or 3D objects that can be placed in the surgeon’s field of vision, avoid using other displays in the operating room, and facilitate more accurate alignment between virtual information and physical objects.
This would reduce the need for awkward postures for the surgeon and provide new interactive experiences in surgical planning.