The purpose of medical imaging from the very beginning was to figure out ways to look inside the body and learn what’s going on structurally and physiologically. To that end, physicians used x-rays or performed exploratory surgeries for decades to identify disease or injury. Then came the ultrasound in the 1960s that gave clinicians real-time images of internal body structures using sound waves. Imaging techniques progressed even further in the 1970s with the advent of CT scans and MRI, which are both commonly used today.
It’s my belief that 3D modeling will be the next critical tool used by physicians to not only diagnose, but improve surgical planning, patient outcomes and the education of future clinicians. It has the power to essentially produce exact replications of soft tissue structures, improving understanding among doctors and patients alike. But first, it will take collaboration across the U.S. to make this a reality.
I recently spoke at the American Heart Association-Midwest Affiliate’s Heart Innovation Forum to advocate for imaging techniques that lead to anatomic replication. The Advanced Imaging and Modeling (AIM) team at Jump Simulation has come up with a semi-automated process to convert CT and MRI scans into 3D digital images that can be printed or integrated into virtual environments like augmented and virtual realities (AR and VR). What we’ve learned is that these nearly perfect 3D surrogates of anatomy can’t happen without working to create quality images from the start.
Garbage In, Garbage Out
The old adage “garbage in, garbage out” applies directly to 3D modeling. The standard across the nation for the last ten years has been to quickly produce images that might not have the best quality but lead to diagnosis in an efficient and productive manner. The ability to print or view these images in three-dimensions, though, requires a little more time and effort but leads to discoveries we’ve never seen before.
There is a quality standard that must be met each step along the continuum of 3D modeling translation. If the image is poor – fail. If the segmentation is poor – fail. If the print is poor – fail. If the VR translation is poor – fail. The focus of our cardiovascular imaging efforts at OSF HealthCare is to generate the highest quality images we can attain.
Most recently, we sent a quality focused 3D heart digital file to the incredible engineers at Caterpillar’s additive manufacturing lab. They have a printer that allows us to produce a heart in a soft enough material that can be cut with a scalpel, allowing surgeons to effectively practice on a patient’s heart before surgery. The result was incredible. Not only were we able to practice the surgery before the operation, but we were able to see anatomic detail like never before seen, prompting an entirely new set of possibilities where 3D printing could potentially improve patient care.
Making a Case for High-Quality Imaging Standards
There are many physicians around the U.S who understand the impact 3D modeling can have on surgical planning, patient outcomes and the education of future clinicians. In fact, a group of us are working with the National Institutes of Health and the American Heart Association to create accuracy and quality standards for the Jump Simulation-curated 3D Heart Library, an open-source digital repository of hearts with congenital defects on the NIH 3D Print Exchange. However, I recognize there are still some skeptics out there who don’t understand the value of this technology.
My experience with these models has been that they give surgeons a point of reference they haven’t had before, giving them the ability to make informed decisions before operating on patients. They make viewing anatomical images intuitive across all medical specialties. 3D models give patients and their families a better understanding of procedures they may have to undergo. They also allow educators to easily explain different types of congenital heart disease and what they look like to physicians looking to master the skill of diagnosis or surgery.
Physicians are busy and it’s difficult to put the time and effort into higher quality imaging. However, doing so leads to exact anatomic replications and, in my opinion, is the next big jump in medical imaging surrogacy. It’s going to take clinicians making medical decisions or planning surgery to be impacted by this for the advocacy to come through the clinical community.
What if medical imaging and visualizing human anatomy were as easy, immersive, and intuitive as playing a video game? The latest project taken on by the Advanced Imaging and Modeling (AIM) program at Jump is working to achieve just that. AIM has nearly perfected the refined, manual process of converting traditional medical images (MRI and CT scans) into digital formats, allowing doctors and surgeons to interact with physical three-dimensional anatomic models for medical decision making, pre-surgical planning, and patient education.
In fact, hospitals from coast to coast are sending us medical images of congenital heart defects and other abnormal conditions for 3D printing. When medical decision making is involved, we provide this service for free in hopes of giving physicians and surgeons improved understanding of each patient’s clinical scenario. The ultimate goal is to utilize intuitive 3D tools to improve the clinical outcomes and standardize interpretations of anatomical images across all clinical specialties.
Now, AIM is taking digital formats of medical scans and plugging them into evolving visual technologies, eliminating the need for a physical model and shrinking the time it takes to view a complete 3D image. We believe virtual and augmented reality will revolutionize how radiologists and clinicians look at anatomy.
Physical 3D Models Vs. Digital Models
Just as there was not an existing process for converting medical images into a digital format, there was also not software to translate imaging data to be compatible with virtual environments like augmented and virtual realities. Our engineers at Jump have created a prototype program to view 3D images of hearts and other parts of the human anatomy for the HTC Vive, a virtual reality headset.
What we’ve found is that physical 3D representations of anatomy (3D printed content) can only be viewed in so many different ways. With augmented and virtual reality, there’s more flexibility to immerse oneself into the entire image, and expand viewing capabilities. It’s also scalable because I can send a video file much quicker than I can ship a physical 3D model.
The AIM team recently ran test case scenarios with several different surgeons to collect feedback on their experiences and whether they found viewing anatomy with the Vive beneficial.
“As one who does not have a background in video games and this type of technology, the HTC was fairly easy to grasp,” said Dr. Karl Welke, a pediatric congenital heart surgeon with Children’s Hospital of Illinois. “It puts you into a more intimate relationship with the heart and you can manipulate it in ways that you can’t do with a typical medical scan. As a surgeon, I can understand what I’m going to see in the operating room in much greater detail than I could before.”
Other surgeons found there to be educational value in using the Vive to view human anatomy. “I could have surgical residents view virtual models of cancer and ask them to practice removing tumors before going into surgery,” said Dr. Richard Anderson, thoracic and cardiac surgeon. “This would give residents more experience before operating on a real patient and prevent errors in the future.”
The Future of Medical Imaging
We see a future state where clinicians are truly immersed inside this environment, and interact with it in ways they were unable to before. We also see the potential for bringing people together inside virtual and augmented realities for group learning and comprehension.
I’ve been doing pediatric cardiac congenital MRI for nearly nine years, and I strongly believe that viewing medical images using immersive, visual technologies is not a fad. It’s the direction of the future. We are putting tools in clinicians’ hands that they didn’t realize they needed and helping them fulfill their potential in medical decision making with higher efficiency and quality so that we can improve health care.
It was more than two years ago when I got wind of Jump purchasing a 3D printer. As a pediatric cardiologist, all I could think of is how this tool could revolutionize imaging of the tiniest of hearts and change the course of a patient’s life.
Jump ran with this idea of converting 2-dimensional images of the heart into exact 3D printed replicas that surgeons could hold in their hands and utilize for surgical planning. These models have improved the surgical outcomes for many patients, and led to the creation of the National Institutes of Health 3D Heart Library.
Our imaging and modeling methods have grown significantly since the first heart was printed. We now have a team of engineers at Jump and University of Illinois working to advance the diagnostic effectiveness of imaging tests around the world.
AIMing for a Better View
The Advanced Imaging and Modeling (AIM) program at Jump is improving the exact replication process of anatomical structures using emerging 3D technologies.
There’s a solid foundation at Jump to print pediatric hearts and other hollow organs using a refined manual process. But how do you effortlessly look inside the solid structures of the brain, liver, or even the pathology of tumors? Solving this issue could be a major breakthrough in healthcare.
Here’s a great example of the work we’re achieving.
This is a 3D model of a patient with lung cancer observed over four points in time. It was created in collaboration with Dr. Beth Ripley from the University of Washington. The yellow represents the cancer within this person’s lung. The thing I didn’t appreciate until this rendering was made is how invasive this cancer is and how it wraps around the organs. The result of this work is improved understanding and an incredible tool for cross-specialty communication.
2D vs 3D
Doctors all over the world are currently working with so-called “surrogates of anatomy” from CT and MRI scans. These images have a 3D data set but are viewed within the scope of 2D—leaving an opening for different types of interpretation.
Radiologists spend their whole careers looking at complex 2D images with the goal of conveying that information to clinicians who don’t have the same background. In many cases, specialists making medical decisions have to walk through these images several times to fully understand them.
This is where new 3D technologies such as augmented and virtual realities and holographic displays come into play. 3D modeling can make viewing anatomical images intuitive across all clinical specialties leading to better diagnoses, surgical planning, education, and outcomes. The simplicity of these models also allows for better communication with families.
The overall goal of AIM is to create a future state where clinicians can interact with human anatomy and pathology in a way they’ve never been able to do before. There’s so much untapped potential for improved understanding with 3D modeling. I look forward to a new reality where 3D analysis of medical images outweighs 2D—changing healthcare outcomes for the better.