Rating Surgeons' Performance: It's Complicated
ProPublica, a non-profit investigative journalism organization, recently published its Surgeon Scorecard, rating the individual performance of nearly 17,000 surgeons performing one of eight elective procedures on Medicare patients nationwide. The scorecard compared surgeons' performance based on their complication rates. The purpose of the report was to provide greater transparency about surgeons' performance with the ultimate goal of reducing medical errors. We asked our U-M experts to weigh in on the implications of the ratings and what they might mean for surgeons and patients.
"It is notoriously difficult to rank individual surgeons — no matter what method you use — and the reason for that is when you're trying to measure performance, you usually look to certain outcomes. To have a good measure, you need an adequate numerator and adequate denominator, and both tend to be small when you look at individual surgeons. There are very few procedures that are common enough to measure individual surgeon performance. If a surgeon does 10 or 20 cases a year, you're not going to have a large enough sample size to measure their performance with any degree of accuracy or precision … When you have surgeon-specific performance reports, what you need to understand is that the outcome of an operation isn't entirely due to the surgeon. It's a team sport and there are a lot of important things that the system brings to bear on caring for individual patients. This focus on rating individual surgeons loses the idea that it's a team effort to produce the best outcomes."
Justin Dimick, M.D.
George D. Zuidema Professor of Surgery
Professor of Health Management and Policy
Member of the Institute for Healthcare Policy and Innovation
"Because most surgeons perform a relatively low volume of any individual case type, it is better, and statistically more meaningful, to compare surgical quality at the aggregated hospital level, where the numbers are higher. And apart from the numbers issue, the data itself has to be reliable and audited for accuracy. The Michigan Surgical Quality Collaborative is a Blue Cross/Blue Shield of Michigan supported group of 73 large Michigan hospitals. It very carefully validates its comparative data, reports its conclusions back to the surgeons and hospitals, and always uses the philosophy of 'measurement to improve, not to judge.' Overall quality has improved dramatically using this approach, and this is far better than trying to target poor individual performance using inaccurate and misleading information. Viewing surgeon performance as a commodity, like purchasing a dishwasher, diminishes what may be one of life's most important decisions."
Darrell A. Campbell Jr., M.D.
Professor of Surgery
Director of the Michigan Surgical Quality Collaborative
"In a field like neurosurgery, with so many different types of operations surrounding complex anatomy like brain stem and cerebral vascular systems, it's very difficult to analyze the quality of a surgeon's performance. But I do think, inevitably, we're going to be judged more and more by outcomes, and we're going to have to help people define what those outcomes should be — whether we look at rates of infection or patients being brought back to the operating room or the neurologic deficits after surgery. Those are very difficult things to quantify and, most importantly, to be able to provide information about in the context of the whole patient ... I think the most important thing we have to recognize is that the accuracy of the data that's provided needs to be clear. One of the worst things that can happen is when assumptions are made about data or poor reporting of data such that patients and physicians can't communicate effectively about truths."
Karin M. Muraszko, M.D.
Julian T. Hoff Professor and Chair, Department of Neurosurgery
Professor of Pediatrics and Communicable Diseases
Professor of Plastic Surgery
"While we hope that patients will use and understand the many different rating systems that have emerged over the last decade, the reality is most patients do not. A large proportion of patients are going to choose their surgeon or hospital without using or knowing about such rating systems. Patients have been shown to rely on factors such as primary care provider referral, existing physician networks, word of mouth, geographic convenience/availability and insurance restrictions. There has also been recent work demonstrating that the various rating systems currently being used to measure hospital quality are often incongruent, which highlights their unreliability and can only serve to confuse patients, providers and payers. Ultimately, while we hope that patients will seek out this information, we have an ethical responsibility to provide reliable and accurate information."
Amir A. Ghaferi, M.D. (Residency 2012)
Assistant Professor of Surgery at the Medical School
Member at the Center for Healthcare Outcomes & Policy
Assistant Professor of Management & Organizations at the Ross School of Business
Division of General Surgery Chief at the Ann Arbor VA Healthcare System
Director of Bariatric Surgery at the Ann Arbor VA Healthcare System