Ranking hospitals means constant tweaks and occasional errors in the effort to get it right. The effort is still a work in progress, but transparency and access to better data would help move things along.
A headline this month in The Boston Globe about a VA hospital in Manchester, NH couldn't have been any clearer: "At a Four-star Veteran's Hospital: Care Gets 'Worse and Worse.'"
It was remarkable, given the fact that late last year, the VA had "raised Manchester's quality rating from three stars to four, putting it in the top third of the entire VA system," according to the Globe.
Two days later, US News & World Report announced that it was delaying the official release of its annual hospital rankings after hospitals found errors during a preview of the data.
Related: Hospital Rankings Contradictory, Cryptic, Confusing
Measuring the quality of hospital care is an extremely complicated endeavor, both technically and politically. And the stakes are higher than ever for hospitals now that the Centers for Medicare & Medicaid Services has linked payment to quality.
Hospital ranking programs take that data to consumers, who are increasingly shopping around for health services. But when media reports challenge the analysis of the data, the question emerges: Are consumers losing confidence in hospital rankings?
Brian Kelly, the editor and chief content officer at US News, doesn't think so.
"Our audience has grown," he says. "We have 100,000 people a day coming to us to look at this information. And we feel very good about it. We think we are providing very comprehensive and very clear information for patients making decisions."
The discrepancy in this year's data emerged when hospitals reviewing their US News rankings prior to the public release pointed out "some things that looked out of whack," Kelly says.
His team went back, found a calculation error, fixed it, and rechecked all of the data. The publication date was moved from August 1 to August 8.
Like other ranking programs, US News is constantly refining its methodology.
This year's changes were particularly challenging, Kelly says.
Analysts expanded the definition of transfer cases to be fairer to hospitals that accept high-risk patients. They used dual eligibility for Medicaid and Medicare to account for socioeconomic status in the calculation of mortality risk in 12 specialties. (CMS is also considering using dual eligibility to adjust for social risk factors.)
And, US News dropped two of the six patient safety indicators (PSI) that it used last year. Both were for complications so uncommon they had little impact on rankings. (That move tracks with concerns about the validity of the PSIs.)
The Leapfrog Group
Even those with confidence in one set of quality measures find it's not unusual to find the same hospital ranked highly on one site and not on another.
Leah Binder, the president of The Leapfrog Group, does not think that's a problem. She likes to point out that each ranking program, including Leapfrog's, has its own formula for evaluating quality. Variation in results is expected.
Consumers are used to that, Binder said. She says reads reviews of products on the Amazon website and trusts them more when they don't agree with each other.
"The public isn't frustrated," she says, but "hospitals are frustrated when they see that various reviews differ."
That's because they've been late to the game of reviews and ratings, Binder says.
"[Hospitals are] struggling with the new world. I understand that and I'm sympathetic, but it is the new world and it's not going away. It's time for them to start to lean into it and get better at coping with transparency."
Consumers Union
Lisa McGiffert directs the Safe Patient Project for the Consumers Union, a policy arm of the Consumer Reports organization. (It is best known for ranking electronics and cars, but the group also ranks hospitals.)
She thinks consumer are confused. It can be hard to get it right and they see conflicting results that emerge from different rating programs. Quality review is a complicated process made harder by limited access to good data.
"The state of the art is not refined yet," McGiffert said. "There is a lot of data that is not available. So, what we have available is often not exactly what we would like to have."
At the same time, before the hospital rankings started to emerge, there was little for consumers to go on, McGiffert said.
"Every five to ten years a researcher would do a project on three or four hospitals and they would extrapolate," she said.
US News has been in the ranking business for 27 years. Kelly says that it sees its data as just one of multiple information sources patients and providers can take advantage of when making healthcare decisions.
The site is "not a one-stop shop. It's not Amazon," he says.
"To me, it is part of the consumer journey. The consumer has to be better informed to make decisions. But they can't assume that the decision is going to come from a page on the Internet."
Tinker Ready is a contributing writer at HealthLeaders Media.