Analyzing How Different Patient Groups Respond to Questions About Their Health Care

    • January 24, 2012

The context. Valid, reliable reports of patients' health care experiences are key to the development of measures of quality of care and a patient-centered health care system. Instruments used to measure patient experience must be consistent across people of different races and with different levels of education. Yet, studies have indicated that members of some racial and ethnic minority groups tend to report worse primary care experiences compared with Whites, with variations in educational level controlled. Some of these differences may stem from systematic reporting tendencies and processes used in responding to questions, rather than from actual differences in care received.

From volunteer to professor and researcher. A decade before health reform, Hector P. Rodriguez, PhD, was re-designing primary care at the Permanente Medical Group in the San Francisco Bay area. During college at the University of California, San Diego, Rodriguez had developed an interest in population health sparked by earlier volunteer work at community health centers. A master's program in public health at the University of California, Berkeley, followed. While there, he was drawn in by the "backlash against managed care in the late-nineties and the resulting tension around capitated payments, the patient experience, and physician incentives." A field placement at Permanente landed him in the world of primary care delivery.

After receiving his master's degree, Rodriguez joined Permanente full-time, continuing to focus on issues related to practice re-design and performance measurement. He knew that "these would be major points of influence in the future and that brought me there."

Of particular concern to Rodriguez were the effects of practice changes and performance measures on vulnerable populations. "Clinicians who serve vulnerable patient populations worry about measures of their performance reflecting the differences in their mix of patients," he says. "A lot of initiatives that compare provider performance don't really account for these differences in a robust way. There is tension in the field about the unintended consequences of paying for performance and publically reporting clinical performance information because of this concern."

The experience at Permanente became the foundation for Rodriguez's future research. "While we want to make health care delivery more effective, we don't want to penalize clinicians for caring for vulnerable patients," he says. "We want to help them manage their practices and improve performance. I am drawn to trying to understand how more equitably to compare the performance of clinicians who are trying to do the right thing."

Rodriguez knew that the Permanente Medical Group was "ahead of the curve" on these issues. The practice had integrated delivery along with reporting systems to track key primary care processes—and could use that information to improve patient care, something that many practices throughout the country were not positioned to do. He began to consider a move from management and administration into research.

"Knowing that primary care is the Achilles heel of the American health care system," he says, "I was interested in promoting research in quality improvement and performance measurement that could make an impact on health care practices as well as specifically on care for vulnerable patients. I thought a research career would allow me to influence the field more broadly than I could within the walls of one organization."

A doctoral degree in health policy and medical sociology from Harvard University was the next step, followed by an assistant professor position at the University of Washington School of Public Health. While there, Rodriguez learned about RWJF's New Connections program through a former Harvard classmate, RWJF Senior Program Officer Debra J. Perez, PhD, MPH, who leads the program at the Foundation.

New Connections: Increasing Diversity of RWJF Programming brings new perspectives to RWJF grantmaking by supporting researchers from historically disadvantaged and underrepresented communities. These talented early- and mid-career investigators can be isolated from others with similar career trajectories and overlooked for funding. Launching a successful research career is demanding in most cases. For the researcher from an underrepresented group it can be daunting.

New Connections scholars are in their early- or mid-careers and are members of ethnic or racial minority or low-income communities, and/or the first in their family to receive a college degree. The program offers research funding as well as career development and mentoring. See the New Connections Progress Report for more information.

An unsuccessful applicant ... at first. Rodriguez did not receive an award after his first application but attended the annual symposium at the program's invitation. "While there, I learned more about the program and this created motivation and camaraderie in ways that I hadn't experienced before," he remembers. "I thought it was a good investment on their [New Connections] part, to cultivate applicants who weren't successful. It was a signal that the program was really invested in us. I also learned how to be attentive to Foundation priorities and make a better case."

Rodriguez re-applied and received an award as a member of the 2009 cohort. He started work on his New Connections project shortly after joining the faculty of the University of California, Los Angeles (UCLA) School of Public Health.

His research: understanding how different groups of patients respond to questions about their health care. For his New Connections research project, Rodriguez studied sources of "differential item functioning"—situations where people from different groups (e.g., racial, ethnic, gender) with the same level of the trait measured (such as quality of patient care) have different probabilities of giving a particular response to the question. "A negative reporting tendency—or any different reporting tendency—may stem from various factors," explains Rodriguez. "It may be related to education, culture, age, or cognitive ability. These are all sources from which differential item functioning can result."

Rodriguez investigated the degree of differential item functioning on the Clinician & Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS) survey. The CG-CAHPS is widely used by physician practices and clinics to ask patients about their recent experiences with clinicians and their staff. It is also used in public reporting and in pay-for-performance programs.

Some racial and ethnic minority groups tend to report more negative health care experiences than White patients report. Knowing the extent to which these reporting differences reflect actual differences in care, versus systematic differences in the way different groups tend to respond to the questions asked, is key to ensuring that the measurement of clinician performance is fair. Responses to an item by individuals with the same underlying trait should, on average, be the same, regardless of demographics.

Using CG-CAHPS survey data from a total of 12,244 commercially insured patients of eight medical groups in southern California, Rodriguez, with colleague Paul K. Crane, MD, MPH, from the University of Washington, examined multiple potential sources (including race, ethnicity, age, gender, primary language at home, and others) of differential item functioning for the CG-CAHPS measures. By considering the cumulative effect of differential item functioning across multiple sources Rodriguez and Crane could ascertain whether it was fair to compare the different experiences of different demographic groups.

Rodriguez found that the overall impact of differential item functioning in the CG-CAHPS survey was minimal, with the survey functioning similarly across patients of diverse backgrounds. In an article published in the December 2011 issue of Health Services Research, he and Crane conclude that "previously documented racial and ethnic group differences likely reflect true differences rather than measurement bias."

The impact of New Connections: an enhanced research career and a community of like-minded colleagues. During his PhD studies at Harvard, Rodriguez was exposed to some of the top thinkers in the health services research field, an experience duplicated and improved on by New Connections. The RWJF program has given him close access to a similar level of research leaders, yet in a "safe setting where our interests as underrepresented investigators are more aligned with each other so we understand the motivations of one another in ways that other colleagues may not—and we don't have to explain.

"For folks doing research focused on minorities and vulnerable populations, it's a space where we can discuss those interests without questioning the underlying motivations of why we're in this work. The community that is created—and that safe space for engaging with top thinkers—are highly valued."

Since New Connections, Rodriguez has received a $3 million grant from the Agency for Healthcare Research and Quality (AHRQ) to compare two models of care and their impact on primary care quality in community clinics. He has also received a Public Health Services and Systems Research grant from RWJF, which focuses on local public health systems and the organizational factors that influence population health. With colleagues at the VA Center for Healthcare Provider Behavior he is also evaluating the patient-centered medical home approach to primary care in the Veterans Affairs Healthcare System.

The New Connections project, the first multi-year research grant on which Rodriguez was principal investigator, made him more attractive, he believes, to other funders. "Even if you have a track record of first-authored publications, not having been a principal investigator on a substantial grant is a hindrance to getting funding as a principal investigator," he says. "The New Connections grant signaled to other funders that I was capable of leading a research project. Despite funders saying that they want to be innovative, they generally fund predictable projects that are incremental in their contribution. This tends to favor older, established nonminority investigators. New Connections is a way to equalize the field."

Looking ahead. Rodriguez is a very busy person. He received tenure in July 2011 and is now associate professor in the Department of Health Services in the UCLA School of Public Health. At about the same time he became associate director of the UCLA Kaiser Permanente Center for Health Equity. In this role, and as part of his work under a Clinical and Translational Sciences Institute award at UCLA, he is helping to facilitate the development of practice-based research networks in southern California. For his contributions to health care management research, he was also awarded the prestigious 2011 Jon D. Thompson Prize for young investigators by the Association of University Programs in Health Administration.

"My future work focuses on partnerships and collaboration, and the data and research that can stem from that integration," Rodriguez speculates. "An example is delivery systems research, which is the focus of the AHRQ grant—comparing systems or models of care and the impact on patients' experience of quality and cost. Looking at delivery system research in the context of these practice networks can provide a diverse range of settings to compare."

The opportunity to lead a research project as a junior faculty member, the training and skills development, and the exposure to research leaders and colleagues across the country afforded by New Connections have all contributed to his career growth, Rodriguez believes. "The impact has been tremendous," he says.

RWJF perspective. "The Foundation has a strong commitment to diversity and recognizes the vital role diverse perspectives have in creating solutions and innovations to address today's most pressing health and health care issues in the communities we serve," says RWJF's Perez. The New Connections program is designed to expand the diversity of perspectives that inform RWJF program strategy and introduce new researchers to the Foundation, while providing support and professional development opportunities to a network of more than 1,200 scholars representing historically disadvantaged and underrepresented communities.