Are Current Patient Satisfaction Surveys Good Enough?

Baylor research suggests CAHPS provider surveys are not useful. A new questionnaire could provide deeper feedback.

Patient satisfaction is top of mind these days, and many health systems devote substantial financial resources and effort to determine whether patients believe their physicians are providing good service, often using the scores in quality improvement initiatives. It may be time to take a deeper look—not at the results—but at the survey itself.

Research from Baylor University in Waco, Texas, indicates that some of the questions asked on Consumer Assessment of Healthcare Providers and Systems (CAHPS) provider surveys may produce data that's of little value, according to a study published online June 7 in Psychological Assessment, a journal of the American Psychological Association.

CAHPS Scales Feature a Flawed Format
 

"Our results showed that the CAHPS scales were quite problematic," says Keith Sanford, PhD, professor of psychology and neuroscience, Baylor College of Arts & Sciences.

In short, the items researchers examined allow respondents to select an answer from a four- or five-point scale, with options ranging from "poor" to "excellent," for example, or from "always" to "never." Previous research demonstrates that this format skews results, Sanford says. Most respondents choose the top response for each item and fewer than 5 percent select the last option.

While such a scale can identify "highly disgruntled" patients, researchers wrote, it does not make reliable distinctions between patients who have marginally acceptable experiences or those having extremely positive ones.

"I'm calling attention to some major problems with the CAHPS instruments, but they're not worthless," says Sanford, a scholar of psychometrics who develops assessment instruments. "[The CAHPS surveys] do discriminate between people who are highly disgruntled and those who aren't. That's, at least, a start."

A New Approach
 

The study went much further than casting shade on current survey instruments; the team created a new tool to gauge patient satisfaction with physician consultations. The research suggests it works better than existing CAHPS tools—on the tested items.

The Medical Consultation Experience Questionnaire (MCEQ) developed by Baylor measures two aspects of patient experience—"alliance" and "confusion." These factors address the physician-patient interaction, but not the results of treatment.

Alliance relates to perceptions that the practitioner is committed, competent, and dedicated to understanding patient desires and viewpoints. Confusion is a mindset providers can prevent in patients by engaging in a good exchange of information.

The questions in the new survey are different from CAHPS, in that several offer a basis of comparison to a "typical practitioner," and the response options are more specific with no scales. One sample question: "Compared to a typical practitioner, how much do you feel like you have a warm and comfortable relationship with this practitioner?" Respondents can select from the following options:

• Less than typical
• Typical
• Maybe slightly more than typical
• More than typical
• Much more than typical
• Amazingly more than typical  

The study validated the new questionnaire, confirming it enables a wider range of measurements with more extensive and specific responses. Also, rather than identifying only the most dissatisfied patients, it delivers an assessment regarding whether patients feel an alliance with their provider, and whether they felt confused after the encounter.

The Baylor team is making the new survey available at no charge to interested parties, and it is also included in the full text of the study, which is available for purchase online.

The Study
 

Researchers examined:

• Six items (questions and response options) from the CAHPS that assesses doctor-patient relationships
   
• Two items from the Child CAHPS that focus on parent ratings of the physician’s relationship with the child
   
• While not part of the formal study, Sanford says the hospital CAHPS survey has three items assessing doctor-patient relationships that are "quite similar to the items we tested"

Researchers first conducted a series of studies to clarify problems with the numbered scales used in existing CAHPS questionnaires. Then, the new survey was developed in a series of seven preliminary studies (with 758 participants) and tested in three subsequent validation studies.

"Medical Consultation Experience Questionnaire: Assessing Perceived Alliance and Experienced Confusion During Medical Consultations" was funded, in part, by a grant from Baylor College of Medicine. Working with Baylor University psychologists were physicians with Texas Children's Hospital and Baylor College of Medicine.

CAHPS Survey Development Process
 

CAHPS surveys were developed by The Agency for Healthcare Research and Quality, part of the U.S. Department of Health and Human Services. According to the Centers for Medicare & Medicaid Services website, "CAHPS surveys follow scientific principles in survey design and development. The surveys are designed to reliably assess the experiences of a large sample of patients. They use standardized questions and data collection protocols to ensure that information can be compared across healthcare settings. CAHPS surveys are developed with broad stakeholder input, including a public solicitation of measures and a technical expert panel, and the opportunity for anyone to comment on the survey through multiple public comments period through the Federal Register. Finally, many CAHPS measures are statistically adjusted to correct for differences in the mix of patients across providers and the use of different survey modes."