The problem. As schools have come under increasing pressure to improve student achievement on state and national tests, children's opportunities to play and run around during the school day have dwindled. Yet, a growing body of research is finding that reducing recess may have a negative impact on classroom behavior and readiness to learn.
Playworks: Re-introducing play into schools. Playworks is a national nonprofit organization that sends trained, full-time coaches to low-income, urban schools. The coaches lead organized play and physical activity during recess, guide games, provide leadership development during class times, and run tutoring and physical activity programs after school. As of the 2010–2011 school year, Playworks was bringing play to 250 schools in 15 cities throughout the country.
In 2005 and again in 2008, the Robert Wood Johnson Foundation (RWJF) provided funding for Playworks to expand its program into new schools. An evaluation of the Playworks program is part of that effort. It is important to note, says Laura Leviton, PhD, RWJF's senior advisor for evaluation, "Playworks is a place-based, randomized experiment, with the schools that did not win the 'lottery' this year put on a wait list. This is an ethical decision because there are not enough slots to fill the demand for the program."
The Playworks evaluator. When seeking an evaluator for the Playworks implementation, RWJF staff identified the John W. Gardner Center for Youth and Their Communities at Stanford University and named Milbrey McLaughlin, PhD, EdM, principal investigator. "Milbrey is a real star in the evaluation of the implementation of educational programs," said RWJF's Leviton.
McLaughlin asked a colleague, Rebecca London, PhD, senior researcher at the Gardner Center, to be the co-principal investigator and day-to-day leader of the evaluation. London brought training in both implementation and quantitative research methods. Leviton describes her as a "bridge" between the two and "devoted to youth development."
London received her doctorate in Human Development and Social Policy from Northwestern University in 1996, the year that welfare reform passed and TANF—Temporary Assistance for Needy Families—became the federal funding mechanism for services to poor families. She joined Berkeley Policy Associates and did a series of state-level evaluations of TANF. This was a time that, she says, "was really exciting—there was so much policy interest at the state and federal level."
With the advent of the Bush administration, the research dollars and focus of TANF shifted to marriage promotion, a direction that London did not find as compelling as other ways to help low-income women. She began to focus on low-income families and issues related to children in those families.
After a professional move to the University of California at Santa Cruz, London became involved in a study of the racial digital divide (the race-based differences in children's access to computers) that analyzed afterschool technology programs. What she found most interesting was that while "technology was what brought the kids in—and they were learning computer skills—that wasn't why they were coming. They were coming because there were all these great things about the afterschool programs and they were having meaningful relationships with adults." London learned that her observations were consistent with the literature on afterschool programs. "This was the beginning of my interest in youth development and low-income kids," she said, "and is what brought me to the Gardner Center."
Step one: Evaluating Playworks' implementation. London and McLaughlin focused their analysis specifically on Playworks implementation in eight schools in the San Francisco Bay Area. "We were not looking at the impact of the program, but were trying to understand the conditions under which the program is best implemented," said London.
Playworks operates in many different schools, with many different kinds of students. Although in the midst of a major expansion, the organization did not have a clear understanding of the conditions that would lead to the best implementation.
London approached the evaluation with an eye toward helping Playworks plan for a large randomized trial of the program and its impact on children and schools, also funded by RWJF. "We thought if we could give them a flavor of what leads to the best quality implementation and what the effects might be on the school environment," said London, "that would give them information they needed for later."
London and McLaughlin settled on four research questions as the basis for the evaluation:
- What variations in the school environment lead to different conditions of implementation?
- How is the program received by students?
- How is it received by teachers?
- How is it affecting the school climate overall?
London led a research team in three rounds of data collection at six schools newly implementing the program. In the fall semester, just after Playworks began at each school, they interviewed teachers, principals, coaches and students and did "a ton of observation," said London. They also conducted focus groups with the fifth grade "junior coaches" (students chosen to assist the recess coaches) and surveyed fifth graders. They continued with observation in the winter months and repeated the fall activities in the spring, along with a teacher survey. "We used a lot of different methods," said London.
At two schools that had longer experience with Playworks, London and her colleagues did just one round of data collection. They interviewed teachers, principals and coaches and they observed recess and class game time. "We wanted to see what the program looked like when it was really ensconced in the school," said London. "We wanted to know what teachers thought about it—not in the first year, when they are still unsure about it—but after five years of being at the school."
London is satisfied with the evaluation design. "We absolutely designed the right study," she said. "We were able to answer all the questions and then some."
Developing a theory of change. Part of any evaluator's portfolio is a theory of change, a tool that articulates how a series of steps can lead to a desired goal. As they reviewed the collected data, London and her staff developed a Playworks theory of change, which describes how various inputs (game rules, positive reinforcement, etc.) to Playworks components (recess, glass game time, junior coach program) influence short- and long-term outcomes.
The theory of change helped London decide what to say about the implementation and how a good implementation would lead to the goals established by Playworks. "The theory of change model is very useful because it helps you map out how to go from the problem being addressed to what you want the school to look like at the end," said London.
Key evaluation findings. London and her colleagues prepared Playworks Implementation in Eight Bay Area Elementary Schools: Final Report describing the "one cohesive story to be told" about Playworks.
The evaluation of Playworks implementation in the Bay Area schools showed a range of positive findings:
- Recess was more structured and students more engaged within just a few weeks of implementation.
- The junior coach position offered significant leadership opportunity for older students who led games, helped younger students resolve conflicts and served as role models.
- Older students, particularly girls, were less likely to participate in the recess games.
- Several factors were critical to successful implementation:
- Early teacher training fostered teacher buy-in of the program.
- A strong coach was key. Coach turnover had a potentially negative impact that could be mitigated by strong support and communication by Playworks staff.
- Factors that coaches were unable to influence, such as school recess schedules and teachers withholding recess to discipline students, challenged implementation.
- Respondents reported reduced conflict, improved conflict resolution and an improved sense of safety at recess.
Based on their findings, the evaluators recommended expanding coach and school staff training, the identification of games appealing to older girls, alternative disciplinary methods to replace the taking away of recess time, and a focus on minimizing coach turnover.
The benefit of the evaluation. "The people at Playworks are really enthusiastic about their program and invested in their goals," said London. "They want to make it as good as it can be." The evaluation showed that each school had implemented the program quickly and with fidelity. "At every single school we went to, the program—with its four or five different components—was in place. Even only three or four weeks into the school year, their program was in place. I think that was incredibly validating to Playworks leadership."
Next steps: The randomized trial of Playworks. RWJF has funded Playworks to expand into 650 schools in 27 cities by 2012. RWJF is also supporting a team at Mathematica Policy Research, Princeton, N.J., led by Susanne N. James-Burdumy, PhD, to conduct a randomized trial of Playworks in 32 schools. Researchers are studying the effects of Playworks on school and classroom climates, bullying, physical activity and educational outcomes such as absenteeism and achievement.
London and her Gardner Center colleagues at Stanford will continue the work initiated in the Bay Area evaluation with the schools in the randomized trial. "It will allow us to say that the conditions at this school led to a weaker implementation, whereas conditions at this other school led to a stronger implementation," said London. "We will look at what it is about the coach's characteristics, the timing of training, how bought-on the teachers are, the physical layout of the school—whatever it is that affects implementation."
RWJF perspective. "This evaluation was an ideal situation for Playworks and for the Foundation," said RWJF's Leviton. "It was highly constructive and highly useful. The findings—such as the impact of school disciplinary policies and coach turnover on implementation and the less full participation by older girls—are helpful for the program as it expands. This evaluation gives us confidence that we are investing wisely in the much larger randomized evaluation," she said.