Aligning Forces for Quality: Program Evaluation 2006 - 2016

About RWJF's Aligning Forces for Quality (AF4Q)

Aligning Forces for Quality, a national “signature effort” of the Robert Wood Johnson Foundation (RWJF), was launched in 2006. The program was designed to help communities across the country align health care forces to improve the quality of health care of 37 million people, providing models for widespread national reform. Over the course of a decade, RWJF’s investment in AF4Q provided unprecedented funding and expert technical assistance to 16 multi-stakeholder health care coalitions (“alliances”) composed of three groups: 1) those who get care; 2) those who give care; and 3) those who pay for care—to work together to lift the overall health care of their communities.

AF4Q Goals / Objectives

In each AF4Q community, a regional alliance of doctors, patients, consumers, insurers, and employers worked collaboratively to transform their local health care system. Lessons from these transformations would then be used to develop national models for reform. Alliances were tasked with addressing five “forces” to enhance quality while reducing costs:

1. Performance measurement and public reporting
2. Quality improvement
3. Engaging consumers in their health and health care
4. Reducing health care disparities
5. Reforming payment
   

Evaluating AF4Q

The AF4Q evaluation was led by RWJF’s Senior Adviser for Evaluation, Laura C. Leviton (2005–2008), and Senior Program Officers Claire Gibbons (2008-2014), and Carolyn Miller (2014–2017). Dennis Scanlon, PhD, Director of the Center for Health Care and Policy Research at the Pennsylvania State University oversaw the evaluation team comprised of researchers from the Pennsylvania State University, the University of Michigan, the University of Minnesota, Northwestern University, and George Washington University. Evaluating this 10-year effort, the evaluation team examined the AF4Q program design and implementation as the national health care environment was undergoing ‘tectonic change,’ including the passage of the Affordable Care Act (ACA) in 2010.

Evaluation Methodology

The AF4Q evaluation was a longitudinal mixed-methods program evaluation (i.e., multiphase triangulated evaluation) of 16 grantee “alliances” from across the country. RWJF continuously adjusted their approaches and program goals as a result of the evaluation team’s findings, monitoring efforts of the AF4Q national program office and RWJF, and changes in national health care policy. Results were shared in journal articles, reports, briefs, presentations, (many available on rwjf.org), and social media posts over the course of the evaluation in the hopes that sharing lessons learned early on would allow those working in other health and health care settings to begin applying them to improve services within their own communities.

Key Evaluation Findings

• Implementing and aligning activities across all five “forces” proved difficult for any one AF4Q community.
• While collaboration among stakeholders to improve systems is conceptually appealing, there are many challenges to sustaining regional relationships, though AF4Q highlighted some promising models.
• Over time, the AF4Q regions showed no major differences in improvement rates in quality and health measures examined, compared with control communities. However, all communities trended toward improvement.
• Engaging consumers in reforming the system proved more difficult than expected, but the conversation has been elevated. The importance of transforming the system “by patients and for patients” is gaining traction.
• Sustaining quality and price transparency at the community level is challenging. Important questions moving forward include: Who will fund and produce this information, and who receives the most value from it?
• Disparities reduction activities proved challenging but happened with more intensity in communities that had long-established relationships with community groups representing diverse and underserved populations.
  

Lessons Learned

With the exception of a small proportion of outcomes, there were no major differences in the rate of longitudinal improvement in AF4Q communities, compared with control communities, on quantitative outcomes related to the triple aim as defined by the Institute for Healthcare Improvement:

• Improving the patient experience of care (including quality and satisfaction);
• Improving the health of populations; and
• Reducing the per capita cost of health care.
  

Although the majority of the measures improved in both AF4Q and non-AF4Q communities, there were some exceptions to this improving trend, most noticeably in the cost of care and population health. There was also considerable heterogeneity across communities in terms of programmatic areas and the scale and scope of interventions in these areas. Although a number of AF4Q alliances implemented robust interventions in specific areas—often advancing strategies useful for others in the field—no AF4Q alliance pursued and aligned all five AF4Q programmatic areas in a robust way. In addition, while all alliances were able to garner the participation of multiple stakeholders initially, sustaining this participation and securing new sources of funding after RWJF support ended proved challenging for many alliances.

Conclusion

Although the AF4Q program did not attain the ambitious community-level changes predicted by RWJF at the program’s outset, it did produce pockets of success on some dimensions for particular alliances. A number of factors explain the less-than-expected impact of the AF4Q initiative on community health and the observed variation in alliance sustainability and intervention strength. These include: differing acceptance of the AF4Q initiative’s theory of change; variation in the experience and capacity of the alliance communities selected for the program; differences in alliances’ local health care market context; and the changing programmatic requirements for alliances participating in the AF4Q initiative. The variation in AF4Q program outcomes offers important lessons for those engaged in regional health improvement work.