The challenge. Evaluators and program managers often operate from different frames of reference. Evaluators conduct rigorous studies aimed at advancing knowledge. They share findings via scientific papers published in peer-reviewed journals. Program managers and directors, on the other hand, look for insights and conclusions that reflect their real-world fiscal, staffing and community environments.
Sara Wilcox, PhD, has earned a reputation for integrating these diverse perspectives, thereby both advancing knowledge and improving practice. From 2002 to 2009, Wilcox led a team from the University of South Carolina in evaluating the Robert Wood Johnson Foundation's (RWJF) $17-million Active for Life® program.
An evaluator committed to real-world impact. From the start, Sara Wilcox planned to focus her career on work that would both generate useful knowledge and help people and communities. She says, "I always wanted to apply my research to community interventions. I am interested in exploring how people make changes and how they do that in the communities they live in."
Wilcox's education reflects her longstanding interest in both rigorous research and commitment to real-world issues. After leaving East Troy, Wis., she earned a bachelor's degree in psychology at Northwestern University in Evanston, Ill., and then enrolled in a doctoral program in clinical psychology, specializing in adult development and aging, at Washington University in St. Louis. She completed postdoctoral training at Stanford University in cardiovascular disease epidemiology and prevention.
"While in graduate school, I was struck by how much of an impact one's physical health had on quality of life. There was a lot of evidence that if we could be physically active, we could improve both our psychological well-being and our overall quality of life. But not much was known about barriers to exercise for older adults. That interested me."
Wilcox's interests were backed up by data indicating that many older Americans live unnecessarily sedentary lives, despite evidence that physical activity slows premature aging and chronic disease. As the overall U.S. population ages, physical inactivity among midlife and older people is becoming an even more critical public health concern. In 2000, 13 percent of Americans were age 65 and older. By 2030 some 20 percent will be at least age 65, according to U.S. Census Bureau data.
A case study of evaluation use. RWJF wants to understand how the evaluations it funds are useful to and used by a variety of audiences, not only by people engaged in the program itself, but those beyond it, such as policy-makers. Wilcox and her team's approach in the Active for Life evaluation seemed promising. To learn more, RWJF asked Judith M. Ottoson, EdD, and Diane Martinez, MPH, to conduct a case study, which included interviews with 23 Active for Life stakeholders.
In their report "An Ecological Understanding of Evaluation Use: A Case Study of the Active for Life Evaluation," Ottoson and Martinez concluded that "there was a lot of use—multiple examples, by multiple stakeholders, in multiple contexts. Some of this was 'big' use, e.g., it changed program structure, organizational learning [and] advocacy and leveraged use to other programs."
Laura C. Leviton, PhD, RWJF special adviser for evaluation and the evaluation program officer for Active for Life, agrees. She points to Wilcox's role in skillfully blending rigorous analyses with the flexibility required of community-based interventions. "Sara's personal style made all the difference. She is collaborative and flexible, yet dedicated to a very high standard of quality."
Practicing what she promotes. Wilcox has always been physically active and has managed to remain so despite a rigorous work and family schedule. She runs about three days per week and works out with weights twice per week. "When I got into this line of research, I knew I couldn't promote activity among others without being active myself," she said.
Wilcox has had to make adjustments to keep up her routine. "I have a five-year old and a one-year old child, and that has forced me to make accommodations, but I do get out as much as I can."
Evaluating a real-world intervention: The Active for Life® experience. Wilcox got an opportunity to pursue her research interests not long after she joined the faculty of the Arnold School of Public Health at the University of South Carolina in 1999. Three years later she was selected to direct the evaluation of Active for Life, an RWJF program that aimed to increase activity levels in adults age 50 and older.
Under Active for Life, nine community-based agencies implemented one of two programs—Active Living Every Day or Active Choices. These programs were effective in helping older adults become more active when tested in tightly controlled research environments, but they had not been mainstreamed into diverse community settings. See Program Results Report for details about Active for Life.
The purpose of the evaluation was to assess whether the organizations were able to "translate" research-based physical activity programs into real-world settings so that large numbers of older Americans could benefit from them.
The two-part, seven-year Active for Life evaluation included a process evaluation aimed at understanding how the two interventions were adapted to make them work in community settings and whether the adaptations retained core elements of the original models. This portion of the study was led by Sarah Griffin, PhD, who is now at Clemson University. Their team also asked staff and participants their opinions of the two interventions.
It also included an outcome evaluation, in which Wilcox and her team tracked changes in participant levels of physical activity, body mass index (BMI), levels of depression and satisfaction with body function and appearance. Wilcox also evaluated the extent to which participants sustained improvements over time, after completing either Active Living Every Day or Active Choices.
Research meets reality: Successful strategies and lessons learned. The nine organizations receiving Active for Life grants from RWJF ranged from a hospital system in rural North Carolina, to a YMCA in urban Chicago, multi-service centers in Detroit, public agencies in Ohio and California, and a faith-based community ministry in Memphis.
Wilcox and her team started the evaluation by paying serious attention to the nine organizations and their staffs. "One of the first things we did was visit all of them. This gave me and my colleagues a chance to walk through their physical set-ups, understand their workflows and meet everyone involved. We took time to explain what we were doing and resolve their concerns. We helped them see that our goal was to understand whether laboratory-developed programs could translate into their communities. We were not judging them."
The agencies differed in their experience implementing evidence-based programs and participating in a research study. "This was the first opportunity for the neighborhood organizations to participate in a national research study, and the learning curve was steep for everyone" said Kimberly Voytal-Campbell, an Active Living Every Day project director in Detroit. "We had to balance the demand for accountability to the national program with our commitment to allowing community organizations to experiment, grow and learn."
Throughout the evaluation, Wilcox demonstrated a capacity to overcome challenges that often drive program and evaluation staff apart, such as the burden data collection places on program staff and the need for standard data collection systems across different geographical areas and agencies.
First, she demonstrated flexibility and willingness to change. For example, after visiting the sites, Wilcox and her colleagues modified research instruments and procedures based on what site staff told them would and would not work in their settings. Site staff quickly understood that Sara and her team listened.
Second, Wilcox developed and sustained a strong partnership with Active for Life National Program Director Marcia G. Ory, PhD, MPH, and Diane Dowdy, PhD, deputy director, both at Texas A&M University. Their teamwork modeled the research-program connections Wilcox strove to achieve in her evaluation.
"The national program office convened monthly conference calls, and we participated in those as well. This gave us a vehicle for sharing what each group was learning from the sites and it helped us to participate in resolving problems before they escalated." The evaluation team also participated in conference calls with the program sites to learn about challenges and successes they experienced.
Third, Wilcox approaches her work believing she has as much to learn as she has to contribute. "I gained a lot professionally from working on Active for Life. I learned valuable new skills that I use in other projects. At many levels, I benefitted from this experience.
"Among other things, I learned ways that evaluators can be independent but still develop close relationships. I found it is possible to do both."
Findings with real-world impact. In 2008, Wilcox and colleagues published several articles presenting findings from the Active for Life evaluation. One of those, entitled "Active for Life—Final Results from the Translation of Two Physical Activity Programs," was selected as one of the top 10 most influential RWJF research articles published in 2008. In 2009, RWJF identified it as one of 25 that "had major policy impact, affected our work and thinking, or stood out in some other way."
Wilcox shared three fundamental findings from the outcome evaluation of Active for Life:
Recognition for outstanding contributions. Wilcox's South Carolina colleagues have also recognized her excellence in research and her contributions to communities. In 2007 she received the Arnold School of Public Health Faculty Research Award at the University of South Carolina. And in 2010, she received the University of South Carolina Educational Foundation Faculty Research Award for Health Sciences, given to the outstanding health science researcher at the university.
Continuing a community focus: helping African-American churches promote physical activity and healthy eating. Wilcox continues to shape and study interventions aimed at improving the quality of life for underserved people. Much of her recent research focuses on African-American women living in South Carolina.
She is the lead or co-investigator on several National Institutes of Health (NIH) grants, including one entitled A Partnership to Promote Physical Activity and Healthy Eating in AME Churches. She says "Under this grant, working closely with African-American churches in South Carolina, we designed a 15-month physical activity and nutrition program to be delivered in the churches. We enrolled 1,200 people and are tracking their health outcomes."
RWJF Perspective. "Evaluation is justified only if it is useful," says RWJF's Leviton. Evaluations shouldn't merely produce products—articles or reports of findings—that sit on a shelf. The case study of Active for Life by Ottoson and Martinez provides insights into how to get findings "off the shelf" by examining an RWJF national program through the lens of what researchers call "evaluation use."
According to Ottoson, "questions about evaluation use try to get at whether anyone ever did anything with evaluation products. Did anyone read them? Listen to them? Pass them on to others? Make a program or policy related decision? Change a program activity? Choose one course of action over another? Shift priorities?"
Reflecting on the Active for Life evaluation, Wilcox's leadership role and her contribution to the field, Leviton observes "Sara's Active for Life evaluation is a model for the field of prevention. Yet only rarely are we able to document the kind of usefulness we see in her evaluation.
"The Active for Life evaluation and the work of Sara Wilcox have been a gift to RWJF and to the field."