Joint Commission execs: Most outcome measures are inaccurate

In a new article in the Annals of Internal Medicine, two Joint Commission executives wrote “the criteria for assessing whether outcome measures are accurate and valid enough to use for public reporting, payment, and accreditation are not well-defined.” These outcome measures are used by CMS to determine hospital quality and accreditation status, so their potential inaccuracy is a major problem.

David Baker, MD, MPH, FACP, is a Joint Commission executive vice president and Mark Chassin, MD, FACP, MPP, MPH, is the Joint Commission president and CEO. To tell if an outcome measure is valid, they write, one should determine:

1.    If strong evidence exists showing that good medical care improves the outcome within the measured time period.
2.    If you can measure the outcome with a high degree of precision.
3.    If the risk-adjustment methodology includes and accurately measures the risk factors most strongly associated with the outcome.
4.    If implementing the outcome measure has little chance of causing unintended adverse consequences. 

If the answer to these four questions is “yes” then you have a good outcome measure. Using this criteria, Baker and Chassin then applied them to several outcome measures currently used or proposed for accountability programs:

•    The CMS’ chronic obstructive pulmonary disease, heart failure, stroke, and pneumonia measures
•    The Society of Thoracic Surgeons and New York State Cardiac Surgery Reporting System’s coronary artery bypass graft surgery measure
•    The National Surgical Quality Improvement Program’s (NSQIP) surgical site infection measure
•    The National Healthcare Safety Network’s (NHSN) central line-associated blood stream infection measure
•    The Agency for Healthcare Research and Quality’s (AHRQ) venous thromboembolism measure
•    A proposed international standard set of measures related to changes in physical function and pain after joint replacement
•    The Hospital Consumer Assessment of Healthcare Providers and Systems’ (HCAHPS) survey

Of the measures examined, only three met all of their criteria. Five of the measures, including all the claims-based, 30-day mortality measures, failed to meet one or more criteria.

“The Joint Commission supports the transparency and public reporting of reliable and valid data on quality and has made such information about accredited organizations public for more than 20 years,” Baker said. “However, the nation needs to take a more critical look at outcome measures. We found that most of the national measures did not pass all of the criteria, particularly the mortality measures.”

“If we are going to publicly report outcomes and reward providers who achieve the best outcomes, we must approach outcome measures as rigorously as we did process measures and use extreme caution to ensure that the measures are valid.”