Before Jump Simulation & Education Center, a part of OSF Innovation, became a reality, OSF HealthCare charged the facility’s leaders to come up with a plan to make the six-floor building and its programs self-sufficient within five years of operation. However, there was no real framework for how the health care simulation industry measures return on investment.
It’s easy to calculate the monetary value of simulation-based training from a business perspective (operating costs, revenues gained from events). But it’s more difficult to determine the value of intangibles, such as improving patient safety and employee satisfaction.
The curriculum team at Jump sought out a number of different theories and models around ROI in health care and education. It landed on a concept called “Measuring ROI in Healthcare.” Jump Simulation is the first center to apply this method to its curriculum.
The system was developed by a group of authors, led by Jack Phillips, the Chairman of the ROI Institute and aims to help health care systems determine the value of projects that were previously deemed more challenging to measure. It’s our hope that this model will increase the type of high quality education and simulation that Jump Simulation learners have come to expect. That will lead to improved care for all of the patients OSF HealthCare serves.
Applying the Phillips ROI Process
Jump Simulation has established an ROI Committee made up of representatives from Performance Improvement, Healthcare Analytics, Business Development, Finance and Nursing and Clinical Education. They have the task of placing value on our highest rated educational programs using the Phillips model which evaluates projects in five levels: Reaction, Learning, Application, Impact and ROI.
We used the Phillips model to measure the value of our Central Venous Catheter (CVC) program. The project looked to determine whether simulated central line training could lower rates of complications that are common with this procedure and lower health care costs.
The first level, Reaction, gauges whether learners find certain simulations relevant to their work and whether they intend to utilize what they’ve experienced in their practice. In a post event survey, learners in the CVC program provided positive feedback on the usefulness of this particular simulation.
The next level of the Phillips model, Learning, determines if simulation participants gained knowledge, skills and confidence from their experience. This means learners should be able to demonstrate new procedures in a simulated environment. As part of the data collected for the CVC study, direct observation of learners showed they performed simulated central line insertions correctly. A self-efficacy survey also showed improved confidence among those who participated in the simulation.
Application is the third level of evaluation in the ROI process. We want learners to not only demonstrate they can perform a certain task within a simulated environment; we also want to ensure those who participate in simulation apply that knowledge in the clinical space. A chart review of CVC simulation participants showed an increased percentage of successful line placements for patients within the Medical Intensive Care Unit (MICU) of OSF Saint Francis Medical Center.
The next question we want answered is whether simulation training has a measurable Impact on performance. Data is collected for this fourth level of the model by determining whether the simulation has helped OSF meet a strategic goal such as improved patient outcomes or increasing employee satisfaction. Chart review and billing and coding analysis of CVC simulation learners found that improved outcomes led to patients being able to go home sooner—by a little more than two days.
The fifth and final level of evaluation, ROI within the Phillips model, measures whether a training program is worth the investment. A comparison of residents who didn’t receive simulation-based CVC training and those who participated in the simulation showed a cost savings of about $5600 per patient. That includes the cost of what it takes to implement the CVC simulation.
Beyond that, Jump has also determined it’s imperative to outline the benefits of simulation that can’t be easily converted to monetary value, but are important based on our Missions and Values. This includes improving patient experience, teamwork and employee engagement.
Level 5 ROI
Because of the way the Phillips ROI model is designed, Jump won’t pursue full ROI evaluation with the more than 100 curricula it offers as the process is labor-intensive and expensive. Instead, we will take our highest rated simulations that can be applied in the clinical space through the ROI model.
The goal is to measure ROI on up to seven programs a year. This gives us a chance to identify curricula that need to be redesigned based on learner feedback and other levels of the Phillips model. It’s our vision to go from good to great on all of the education Jump has to offer.