Why Measuring Surgeons Frequently Get the Wrong Results

By Richard A D Jones

What gets measured gets improved. That’s what quality professionals say. However, that requires the ability to measure performance in a meaningful way. This is a problem in hospitals – and particularly with surgery. Unlike in the manufacturing process, these issues with measuring performance have serious financial and human consequences. Simply put, how we measure performance in US hospitals is leading to avoidable deaths and harm.

That’s quite a provocative statement but consider a 2016 report in the Daily Telegraph. It found that one in three heart surgeons in the UK was avoiding high-risk patients because of the way they are measured. Take a moment to consider that. If this is truly the case, it’s not an isolated incident, it’s endemic. It means that people that dedicate their lives to saving others are acting against their nature because of a system of measurement that punishes them for caring for the most vulnerable. The patients that need them the most. If hospitals can’t accurately risk-adjust, how can they make the best care improvements and decisions for their patients? 

So, what’s the problem?

Simple statistical approaches, widely used in hospitals, are good for providing pretty pictures and a false sense of security. They are deeply flawed. Measuring mortality as a percentage of admissions does not consider any differences in the case-mix of patients from one hospital to the next or how this changes over time.  

Consider hospitals in New York. The simple mortality percentage will have increased significantly suggesting a drop in performance that does not factor in the impact of the Covid-19 crisis. That’s an extreme example but measuring mortality and complications as a simple percentage of admissions or procedures is similarly flawed. A UK government report expressly states that metrics [like this] should only be treated as a “smoke alarm” – accepting that these tools deliver significant ‘false positives’.

If we focus on surgery alone, this type of measurement does not account for any underlying physiological factors of the patient that can increase the risk of mortality. There are of course some statistical systems that claim to be “risk-adjusted” but they focus on crudely, high-level markers – like whether or not a patient has a chronic condition. Let’s take diabetes. If a person manages their diabetes well, they are at no greater risk than a healthy patient. Their risk increases if their diabetes is poorly managed, but the risk comes not from diabetes itself: it comes from the foot ulcers, the poor circulation, and all the other side effects that poor management has caused. So really we are talking about more complex, physiological measurements that statistical methods completely overlook.

There is a similar argument for complications/harm and there’s certainly more to life than death in considering hospital performance. In the US, there are sites where you can ‘assess’ the performance of surgeons. ProPublica and Consumers’ Checkbook provide information on the number of procedures and the complication rates but the lack of consideration of the patient’s underlying age, health, etc. will similarly lead to flawed results.  

Where’s the problem?

Many issues leading to mortality and complications occur postoperatively. In places where the operating surgeon is not involved in the post-operative care of the patient, how can they be held responsible?  

This is part of a broader issue as currently, hospital administrators in US health systems can only identify about 10% of the variation that leads to avoidable costs, harm, and mortality. Up to 90% goes undetected by monitoring and reporting systems in a typical hospital and 6 out of 7 instances of variation/harm are undetected even in a system with a reputation for outstanding quality.

So, what should be happening?

In a ‘get it right the first time’, ‘do no harm’, healthcare environment, hospital teams should be supported with the right information, and should be searching to improve the accuracy and quality of the information they use. 

In the context of surgery, the focus should be on measurement of correctly risk-adjusted outcomes that reflect individual physiology of patients – and also the inherent risk of the operation itself – identify issues with mortality and complications and can locate where and why those problems are occurring in the hospital.  

As a final point, consider looking for systems that can be used prospectively to assess the risk of harm and mortality to patients before surgery – meaning you can identify those whose condition should be optimized first, to effectively manage the patient’s individualized real risk factors, not those listed on a pamphlet.

RICHARD A D JONES

President (US) and Chief Strategy Officer, Copeland Clinical Analytics (C2-Ai)

With over 30 years in technology and telecoms, Richard has extensive experience as an entrepreneur, in strategy development, business planning/modeling, and creating commercial implementations for companies. He has consulted with and founded several businesses, across four continents, that have delivered up to 250 times the first-round value. 

He is the only private sector member of a national regulator’s synthetic AI patient record and medical AI software validation project. He is also the roving US correspondent for World Healthcare Journal.

In addition to his work at C2-Ai, he holds positions in an Ai based business and multiple business startups, as well as authoring three books focused on business strategy. Richard received an MBA with distinction from the Warwick Business School.