The Program Being Evaluated
The Foundation's Health Games Research: Advancing Effectiveness of Interactive Games for Health program was designed to build the field and advance knowledge about the intersection of video/computer games and health and health care through a national program that emphasizes applied research and convening.
About the Evaluation
Led by Marjorie A. Gutman, PhD, under contract with Macro International, this evaluability assessment was conducted to determine the readiness of the Robert Wood Johnson Foundation (RWJF) national program, Health Games Research (HGR), for evaluation; to propose options to increase the program’s readiness for evaluation; and to suggest the focus, design, and methods that would produce a useful evaluation. Feedback from the evaluability assessment aimed to help improve program design, implementation and evaluation readiness, and assist in making decisions regarding whether and how to evaluate the program.
Major Evaluative Topics and Questions
The objectives of the evaluability assessment were to examine the HGR national program with regard to the following:
- A plausible, well-developed program design developed on the basis of scientific or political theory, empirical evidence, and/or sound logic, such that experts would agree that the program is likely to bring about the desired outcomes.
- Goals and objectives that are well defined and of a scale and scope to be reasonably achievable from the program being proposed.
- A viable process proposed to implement the program.
- Actual execution of the program process in such a way that it is likely to have theoretically achievable results.
- Readiness for evaluation
- Program information needs are well defined.
- Evaluation design conditions that would allow examination of the questions of interest.
- Available or collectible data to answer those questions at a reasonable cost.
- Receptivity by key stakeholders to fully participate in and partner with an evaluation team.
Summary of Methods
The evaluability assessment consisted of five steps: (1) preliminary interviews and meetings, (2) document review, (3) key informant interviews, (4) refinement of the logic model, and (5) descriptive analysis of available quantitative data.
Knowledge and Impact
The evaluability assessment findings suggest that HGR is, or can be, ready for evaluation with the right supports in place. Given the degree of program design plausibility, feasibility of program implementation, and level of commitment and dedication of the national program office and other stakeholders, it could be cost effective to invest in an evaluation that would provide much needed information not only for RWJF and the rest of the program leadership, but also for the world of health games research and for other Foundation initiatives that are attempting similar field-building approaches.
The evaluation team recommended using an evaluation that uses the working logic model being developed as the underpinning and focuses foremost on responding to the needs of the RWJF Pioneer Team (and the rest of the Foundation) and the rest of the program leadership, while at the same time offering some useful information for the broader research community and field. The evaluation would examine process/strategies implemented, products and outputs, and at least some outcomes represented in the logic model. Careful consideration should be given to the time frame for outcome evaluation, using estimates of time needed to produce the short-term, intermediate, and long-term outcomes. Similarly, careful consideration should be given to resources available for evaluation and the synchrony between those and an evaluation plan.