Spreading Community Accelerators through Learning and Evaluation - Program Evaluation

Recipient of the 2017 American Evaluation Association Outstanding Evaluation Award

About Scale

Spreading Community Accelerators through Learning and Evaluation (SCALE) was a Robert Wood Johnson Foundation (RWJF)-funded initiative led by the Institute for Healthcare Improvement (IHI) along with key community partners (January 2015–January 2017.) SCALE was the first community-based initiative in IHI’s 100 Million Healthier Lives Campaign (100MHL). The 100MHL Campaign aims to accelerate communities (20 ‘pacesetter’ and four ‘mentor’ communities received funding) toward a Culture of Health with the goal of having 100 million people living healthier lives by 2020.

SCALE Goals and Objectives

SCALE’s aim was to build community coalitions’ readiness and capability to improve health, well-being, and equity within the United States. SCALE aligned with RWJF’s vision to promote a Culture of Health for all people by changing social, policy, financial, and community-level structures.

To reach the goal of health, well-being, and equity within communities, SCALE targeted three primary drivers of change:

1. Develop leadership capability to transform and improve within and across communities;
2. Create vibrant relationships and functional networks between communities that accelerate trust, learning, and achievement of a shared goal; and
3. Create and improve an intercommunity spread system.

Evaluating SCALE

The SCALE evaluation team, overseen by Robin Mockenhaupt, senior vice president for strategic initiatives, Laura C. Leviton, former special adviser for evaluation, and Oktawia Wojcik, senior program officer, all of RWJF, was lead by Abraham Wandersman, principal investigator and Jonathan Scaccia, project director. Their team used a formative evaluation approach to understand the contributors of progress and spread toward building a Culture of Health. A key objective of the evaluation was to emphasize evaluation use by having evaluation findings be available in a timely, understandable, and actionable way to help improve the provision of support for enhancing community activities. Evaluation results were gathered continuously, and collaboratively analyzed by all stakeholders, with program changes made based on the findings.

Evaluation Methodology

Along with a collaborative, formative evaluation approach to foster learning and adaptation during the implementation of SCALE, the evaluation team implemented a mixed-methods, multilevel evaluation design to monitor: a) how support was provided to communities; and b) how communities progressed on their community health improvement journey.

Key Evaluation Findings/Lessons Learned

SCALE provided communities with a comprehensive support system involving training, technical assistance, tools, and quality improvement/quality assurance. These supports were co-designed by stakeholders in order to facilitate transformational change along the three primary drivers (see Goals/Objectives above.)

• Community Health Improvement Leadership Academies (CHILAS): in-person, multi-day training sessions designed to foster relationships between communities, build skills in community health improvement leadership, and facilitate learning across communities. Across all four CHILAS, communities highly rated the overall quality and content.

• Coaching: provided by improvement experts to communities and in a group format (Peer Community Teams) were viewed positively with the direct coaching getting higher scores.

• Other support methods like webinars, online courses, and a community health-specific social media platform were also used to facilitate improvement and spread of ideas. These were useful, but not as much as the coaching; also, attendance in the webinars decreased over time (from about 60 attendees in mid-2015 to about 15 by the end of SCALE). The mentor community role was not seen as an effective means of support.

A large and complex initiative like SCALE demands a lot from communities; and the SCALE communities, despite feeling overwhelmed at times, rose to the occasion. Since the design of the support system was taking place at the same time as implementation, there were occasions where communities, like the coaches, felt the need for a roadmap for understanding milestones and expectations. But overall, communities reported significant progress on building improvement capability.

As would be expected in a 24-community study, there was considerable variability in progress. As SCALE evolved, the Community of Solutions skills emerged as a way to organize the key capabilities that communities were expected to acquire.

For more in-depth findings and discussion, please refer to the final report.