Man using virtual reality technology


AR, VR, and 3D Printing Bridge the Gap between Radiology and Surgery at UCSF

4 min read

3D technologies such as augmented reality (AR), virtual reality (VR), and 3D printing provide doctors with more information than they can see with 3D images on 2D displays, making the tools valuable in surgical planning at the University of California San Francisco (UCSF). UCSF hosts one of the premier academic medical centers in the world, providing high-ranking care across many specialties and supporting extensive medical research and innovation efforts. 

In this blog, we sit down with Dr. Jesse Courtier, Pediatric Radiologist, who specializes in utilizing augmented reality for research and patient care at UCSF.

What is the driver for using 3D technology (AR/VR, 3D printing) at UCSF?

At UCSF, we are a referral center for very complex surgeries across a number of subspecialties. As such, the degree of surgical case planning required is especially high. Medical imaging plays a critical role in this planning. We have found that 3D imaging provides surgeons with important additional conceptual information for planning these complex cases that is beyond what is achievable with simple 3D reconstruction on 2D displays. Personally, as a Pediatric Radiologist, the ability to convey complex information from the CTs and MRIs in a way that is clear and understandable to my surgical colleagues is critical. 

3D technologies such as 3D printing and AR, which is my area of focus, provide a powerful tool to help bridge the gap between the 2D world of radiology and the 3D real world of surgery. Further, from the surgical standpoint, finding methods to reduce the overall cognitive load encountered during these complex cases through sophisticated planning is an additional welcome benefit.

What are the main applications for which you are applying AR?

While I see applications for AR across a number of fields, we have found certain subspecialties that are especially amenable to planning with AR. Specialties such as orthopaedics, cardiothoracic surgery, interventional cardiology, liver transplant, and pediatric surgery are among the fields which deal with challenges of high visual-spatial complexity and extensive anatomic variation in their cases. I have made more than 80 AR models at UCSF across a number of subspecialties. Some specific examples include complex pediatric elbow fractures, deformity repairs, large liver tumors, and a very complex congenital cardiac case. 

We are also researching mobile AR technology applications in medical student anatomy small group sessions. In addition, we are also beginning to investigate AR in patient education settings in congenital heart disease. We are excited about exploring the potential for improved patient compliance and overall lower anxiety with an improved understanding of their conditions.

How does AR complement 3D printing?

I believe AR complements 3D printing by allowing for rapid prototyping and iterative model improvement in an economical and environmentally friendly way. Models can be repeatedly tested and viewed at full anatomical scale without regard to some of the constraints of physical models (gravity, thickness, printer size, cost). It also allows for remote collaboration through the sharing of 3D models with colleagues without the need for physical modeling.

What software do you use for preparing medical image data?

I initially began my journey into the 3D world using freeware and off-the-shelf applications. However, I became increasingly aware of the need for more sophisticated software applications as I went further along in creating multiple model types at UCSF. I needed software that was robust, included a variety of features, and had attention to user experience and user interface design. 

At UCSF, through our Center for Advanced 3D Imaging + of which I am a Co-Director, we purchased the Materialise Mimics Innovation Suite, and I have been very impressed with the features and capabilities. Currently, as a final step to optimize for my specific AR app for the Microsoft HoloLens, which we named ‘Radha’ (or Radiology with Holographic Augmentation), I use Blender to add AR-optimized colors, shading effects, and polygon count.

Man using virtual reality technology with another man watching the process on a screen

How has Materialise software enhanced your work?

A current project we’re working on with my colleagues in the ZSFGH Orthopedic Trauma Institute is an assessment of 3D AR models to improve preoperative classification of complex acetabular fractures. We hypothesize these models will allow for a lower degree of inter-observer variability in fracture classification, which in turn guides appropriate clinical management in these cases. We aim to investigate the ability of 3D AR models created with the Mimics software’s thickness analysis tool to determine optimal placement of fixation devices based on the bone thickness.

Could you share a specific case or patient that was impacted by the use of AR?

One particular case was a patient with a very complex congenital cardiac and abdominal malformation that required extensive chest and abdominal wall reconstructive repair. This case was planned between surgeons in pediatric surgery, pediatric cardiothoracic surgery, and plastic surgery. I segmented the case and created a full-scale holographic model including skin surface, bone, heart, lungs, airway, solid organs, and bowel. This model would have been extremely costly and challenging to display with any other method. I reviewed the model with my surgical colleagues in our surgical planning conference prior to the procedure, and it was a success!

Looking forward, what is needed to support the continued growth of this technology in medicine?

I believe continued evidence building on the impact of augmented reality on clinical outcomes such as time under anesthesia, overall operating room time, and time under fluoroscopy will be critical for its widespread adoption. Continued innovation both in AR hardware and software will also be necessary to enable larger, more complex models that are not simply static representations, but animated models that reflect physiologic movements (breathing, heart rate, etc.). This will truly allow for realistic simulation and conceptualization for pre-surgical planning and more.

Share on:

Share on Facebook
Share on Twitter
Share on LinkedIn
Share with Pocket

You might also like

Never miss a story like this. Get curated content delivered straight to your inbox.

Sign me up