Safe, not Brilliant

It was a Friday morning Morbidity and Mortality conference, and Sammy, the chief surgical resident, stood with the other penitents, ready to confess his surgical sins. He described a difficult chest tube placement complicated by injury to the underlying pulmonary artery, necessitating an emergent trip to the operating room for repair. Hearing that, the chairman of the Department looked up, concern bordering on consternation. Already pacing back and forth, awaiting his fate, Sammy lost command of English, his second language. When asked whether he had ever done this before, his response was “I have enjoyed doing this on several occasions.” The chairman’s look was reminiscent of a biblical painting of a very angry god. With that, he uttered the words that would become our unofficial motto, “I am here to train safe surgeons, not brilliant ones.” Tutus non praeclarus – safe, not brilliant. 

Saving Lives, or the Scorecard? 

A recent STAT article noted that the health system, whose flagship includes Mass General Hospital (MGH), has been touting its improving patient safety record. While MGH, “long the top hospital recipient of federal research funds and consistently dominating U.S. News’ lists,” is considered a benchmark hospital, its safety rating, measured by Vizient, a consortium of academic and community hospitals, had declined but has now improved through safety initiatives. [1]

Specifically, the now Mass General Bringham (MGB?) system touts its safety measures, which have saved an additional 1,400 patient lives. The STAT piece also places that claim in a broader institutional context: the system’s expansion into a larger hospital network, including smaller community hospitals that may acquire the “MGH brand” without necessarily reproducing the expertise or outcomes of its flagship centers. It also shares physician concerns about the “corporatization” of medicine and their increasingly diluted voice as administrative roles outpaced clinicians. 

A portion of the improvement came from better communication among clinicians. Termed “huddles,” these brief, structured meetings, whether formal or informal, allow care teams to review safety metrics and frontline concerns in near real time, enabling staff to adjust care quickly. As a physician who worked in both academic and community settings, I found MGB’s embrace of the huddle refreshing, even though the practice arrived more than a decade after many hospitals had already learned its value. I guess that is what is meant by leading from behind. 

MGB has also adopted Epic’s sepsis protocols, which use real-time data from the electronic health record to identify patients who may be developing sepsis. The goal is earlier intervention for a condition that can become lethal quickly. The impact of these early alarm systems on saving lives is mixed, with concerns that false positives waste resources and false negatives lead to unwarranted calm. 

When the Denominator Does the Work

The appeal of Vizient’s data is that it appears more objective than reputation-based rankings. As STAT put it, “Unlike U.S. News and World Report’s list of best hospitals, which considers reputational scores from physician voting, Vizient’s data only looks at outcomes.”

One might be encouraged to believe that Vizient’s outcome data would be more objective; after all, a dead patient is difficult to ignore. Some physicians quoted by STAT described the huddles and real-time metrics as “real harm-reduction outcomes… material changes in our behavior that are benefiting our patients.” 

However, some of the improvements appear to come not from bedside care but from accounting. One hospital clinician quoted by STAT called part of the change “administrative” and even “a sleight of hand.” Consider two mechanisms.

The first way MGH lowered its in-patient mortality was by getting those patients off the books. In medical terms, they increased the percentage of patients moved from the inpatient hospital to the inpatient hospice. Arguably, this is a win for patients, who are no longer subjected to unwanted care and are allowed, by all measures, a more acceptable end of life. It is a win for the health system, as hospice costs are often lower than hospital-based end-of-life medical interventions. And, of course, it is a win for MGH’s metrics. But even when such transfers are compassionate and appropriate, is moving a patient off the inpatient ledger really the same thing as saving a life?

The second way MGH lowered inpatient mortality is reflected in the metric shown in the graph: the ratio of observed to expected deaths. While the numerator in this ratio is an easily measured outcome, the statistical chicanery lies in the denominator, the expected deaths – which require predictive modeling. When expectations equal outcomes, the ratio is 1. When observed deaths exceed expected, the ratio is greater than 1, indicating a significant problem. When the ratio drops below 1, as at MGH to 0.6, we are “saving lives.”  

The problem with predictive modeling is that it can be “gamed,” influenced by documentation and coding. Medicare Advantage has repeatedly shown how risk adjustment can be distorted when patients are coded as sicker than they are, thereby increasing expected costs or risk. A hospital mortality model can be vulnerable to a similar dynamic. It is not unfair to say that MGH has taken a page from that playbook and uses its electronic health record data, already tuned to extract the greatest procedural care (and therefore revenue), to inform (and raise) its risk assessment. The observed-to-expected ratio improves, even if the actual number of deaths has not changed by the same amount. The model can make “lives saved” appear larger than the bedside reality. 

It is not that the metric is necessarily flawed; it is what the metric purports to measure. If we were concerned about predictive ability, a ratio close to 1 would confirm that our predictive model accurately assesses patient deaths. But when we are concerned with saving lives, we assume the predictive model is accurate, an assumption that the health insurers have demonstrated is deeply flawed.

As a physician leader with experience in safety, I can say that patient safety is hard, humbling work. It rests less on slogans and dashboards than on habits, culture, supervision, repetition, and the willingness to ask whether a good-looking number reflects better care. Huddles may help. Sepsis alerts may help. Hospice transfers may be compassionate and appropriate. But none of these should be mistaken for proof that a health system has suddenly learned to save 1,400 more lives.

MGB may indeed be safer than it was. But when “lives saved” depends on moving patients off the mortality ledger or recalibrating who was expected to die, the claim deserves scrutiny, not applause. My old chairman had it right: the task is not to produce brilliant institutions with brilliant metrics. It is to train safe ones.

[1] STAT reported that MGH rose from 96th to 37th in Vizient’s overall quality rankings

Source: Mass General Brigham says it is saving more lives. Some doctors aren’t so sure. STAT