The Program Being Evaluated
As the search continues for answers to address childhood obesity effectively, organizations and communities across the country are experimenting with various strategies aimed at changing children’s environments to reduce the incidence of obesity. The Robert Wood Johnson Foundation (RWJF), the Centers for Disease Control and Prevention (CDC) Division of Adolescent and School Health, Division of Nutrition and Physical Activity, and Prevention Research Centers Program Office, and the CDC Foundation, are undertaking a two-year collaborative project to identify and assess local-level programs and policies that appear promising to improve the eating habits and physical activity levels of children.
The lead evaluators for this program are Laura Leviton, Ph.D., Special Adviser for Evaluation at the Robert Wood Johnson Foundation; Laura Kettel Khan, Ph.D., Senior Scientist for Obesity Prevention and Control in the Division of Nutrition, Physical Activity and Obesity at the Centers for Disease Control; and Nicola Dawkins, principal at ICF MACRO.
Major Evaluative Topics and Questions
The major question was: which of the many programs and policies addressing childhood obesity are most promising and would have a payoff for further investigation? Rigorous evaluation and research studies are costly and time-consuming, and they would be prohibitively expensive if conducted for each and every promising program or policy. In order to guide investments in evaluation and research, this project employed a unique approach to identifying a large volume of promising programs and policies. These were then subjected to several screening procedures to ensure that only those were selected that met stringent criteria. The criteria were: that there be potential for impact; that they were plausible to reach a large portion of the population; and that they were feasible and transportable. The centerpiece of the process was a pre-evaluation assessment method known as an evaluability assessment. Evaluability assessments can be used to determine whether a rigorous evaluation study is feasible and merited for a particular program. Evaluability assessments help avoid premature investment in evaluation studies of programs that have not been adequately implemented and allow evaluation resources to be targeted to programs and policies that are most likely to fill important gaps in the evidence base for obesity prevention.
Summary of Methods
In each year of the project, nominee programs and policies are identified through a systematic search process. An emphasis is placed on policy or environmental programs being implemented in community settings intended to address obesity in children aged 3–17. Priority is given to programs that address low-income populations and ethnic groups that experience disproportionate childhood obesity. A Funder’s Advisory Committee identifies three themed areas. In year one the themes were: access to healthy, affordable food in poor neighborhoods; local school wellness policies as mandated by the USDA; and child care environments (day care and after-school). In year two, the project again focuses on food access and child care settings, but also on comprehensive school health programs, and on the built environment to encourage physical activity. Hundreds of promising programs and policies have been identified and then screened by CDC staff. The Expert Panel selects approximately 10 programs for each theme for an evaluability assessment. Evaluability assessments are guided by selection criteria outlined above.
Each evaluability assessment consists of reviews of program documents followed by a two-day site visit where trained project staff assess program implementation, data collection and program outcomes. As part of the site visit, a limited amount of on-site technical assistance is provided, focusing on topics such as the program’s logic model, substantive expertise, substantive expertise on nutrition and physical activity, and evaluation. Based on the findings of the evaluability assessments, the Expert Panel identifies those programs and policies that show both promise in addressing the issue of childhood obesity and readiness for rigorous evaluation. Project funds currently support one such rigorous evaluation, while other studies of the other programs and policies may be funded by government or charitable organizations.
Knowledge and Impact
Of 282 promising programs identified in year one, 25 were selected for evaluability assessments. The Expert Panel identified 14 of these as ready for evaluation and sufficiently promising to be worthy of evaluation. One innovation was selected for evaluation using project funds. A confidential report to funders detailed year one projects and their outcomes. Three synthesis reports for the projects identified in year one, including food access, school wellness policies and day care and after-school environments, will be released in 2009. Twenty-seven evaluability assessments are currently under way across four thematic areas in year two.
Over a two year period the project examined 458 such innovations; 48 of these were studied further, and 20 were highly promising, in terms of likelihood of producing large population effects and being ready for evaluation; and six were judged as having the highest priority for study.
Evaluation funding has been leveraged for five of the six highest priorities: one by RWJF (New York City day care regulations), four by CDC (two beginning in 2009 and two approved for 2010). The CDC Prevention Research Centers explicitly adopted the remaining innovations for further study.
The CDC is highlighting the information in several ways. There are syntheses in five areas, noting facilitators and barriers to the innovations, including: food access for poor populations; day care and after school settings; the built environment for physical activity; Comprehensive school-based physical activity programs; School district local wellness policies. Additionally, CDC guidance on comprehensive school-based physical activity programs will be issued in 2010, and the University of North Carolina Center of Excellence for Training and Research Translation is highlighting the most promising innovations on their website.
This assessment has resulted in several publications, including:
- a peer-reviewed monograph on the study in New Directions in Evaluation describing a new method in evaluation called the Systematic Screening and Assessment Method;
- several articles on the syntheses developed by CDC; and
- a 2010 article by Dr. Laura C. Leviton, Special Advisor for Evaluation, the Robert Wood Johnson Foundation, for the Annual Review of Public Health on evaluability assessment describing the use of the method.
Adaptations of the method are currently being used in RWJF’s Intimate Partner Violence prevention project in immigrant populations, by a project consultant, Debra Rog, in a federally-funded project on mental health, and by the CDC Division for Heart Disease and Stroke Prevention on cardiovascular health.
The 48 innovations received feedback and technical assistance from the site visitors and CDC obesity prevention experts. A follow-up survey indicated that they appreciated the feedback and the visibility that the site visits offered them, and a large minority used the results to improve the management or further development of their innovations.