Finding the Impact in a Messy Intervention

Using an Integrated Design to Evaluate a Comprehensive Citywide Health Initiative

The Urban Health Initiative (UHI), a national program of the Robert Wood Johnson Foundation (RWJF) was designed to encourage a broad cross-section of the community to work together to see whether it is possible to make measurable improvements in the health and safety of urban youth. This article looks specifically at the design of the UHI evaluation. It highlights the program’s integrated evaluation design, bringing together a theory of change and a quasi-experimental approach, including comparison city usage. By integrating the evaluation design, the authors were able to keep the evaluation design flexible in the early years, enabling the site’s experience to be reflected in the analysis.

This article specifically focuses on the later stages of implementation and program impact. The authors examined differences between program and comparison cities, citing them as impacts only if they were predicted by either program theory, local plans for action or early implementation. The authors found the local implementation fairly consistent with the program theory.

As new approaches to solving old problems become more frequently funded, the authors argue that evaluators must also take new approaches. The new approaches must be focused on using a variety of tools in synergistic ways to strengthen the evaluations of complex initiatives.