Studying What Works, What Doesn't, in High-Risk Youth Mentoring
For 18 months beginning in mid-2000, Public/Private Ventures, Philadelphia, tested the feasibility of evaluating a Portland, Ore.-based mentoring program for high-risk children, called Friends of the Children.
The program employs paid full-time mentors who spend at least four hours per week with no more than eight children, for up to 10 years. Friends of the Children addresses a growing concern that caring and responsible adults are insufficiently involved in the lives of at-risk youth, and volunteer mentors are not sufficient for youth at highest risk.
- Program youth exhibited more delinquent/negative behaviors than did comparison youth. However, program and comparison youth scored similarly in several areas.
- Teachers ranked the program youth as more seriously "at risk" than the comparison youth. According to the principal investigator, however, the comparison youth also exhibited behaviors that would make them appropriate for the program.
- Program and comparison youth were similar demographically.
- Creating a retrospective comparison group of fourth to sixth graders was impossible, since no documentation on them had been saved. However, data from fourth to sixth graders currently in the program could be compared to national samples of youth.
The Robert Wood Johnson Foundation (RWJF) provided a $298,200 grant for the feasibility study between July 2000 and October 2002.
Within the field of youth development, there is growing concern among researchers and practitioners that caring and responsible adults are insufficiently involved in the lives of at-risk youth. That concern has generated great interest among researchers in finding and evaluating promising youth mentoring programs that could serve as models for the country.
Volunteer mentoring programs have demonstrated effectiveness in reducing risk behaviors and improving school performance of at-risk youth. Unfortunately, volunteer mentors are not effective with youth at the very highest risk of delinquency, substance abuse, school drop out and other behavioral problems.
To address the need for more effective mentoring of youth at highest risk, the Friends of the Children program was created in Portland, Ore., in 1993. Friends are paid mentors who serve as trusted and dependable confidantes, modeling successful behaviors and lifestyles to no more than eight children at a time. They spend at least four hours per week with each child, engaging them in one-on-one educational and leisure activities, for up to 10 years.
In 1998, Public/Private Ventures a Philadelphia-based national nonprofit organization dedicated to improving the effectiveness of social policies and programs affecting youth and young adults conducted a preliminary assessment of the program's potential for effectiveness and replication potential.
Their assessment found low drop out rates among both children and adults and concluded that Friends of the Children was a mentoring program worthy of further investigation.
In 2000, 32 Friends served 225 children in 67 schools/neighborhoods in the Portland metropolitan area, and Friends of the Children launched an effort to replicate its work in other cities.
RWJF provided a grant of $298,200 to Public/Private Ventures to test the feasibility of evaluating the program. With RWJF funding, Public/Private Ventures completed the research design for two studies, evaluating both the impact of the Friends program and its replication potential.
The key challenge for the impact evaluation was finding appropriate comparison groups. The Friends' selection process identifies approximately 32 high-risk first graders in each participating school, and then narrows this group to the eight boys and eight girls from each school who receive mentors. Public/Private Ventures sought to determine whether the children not selected could be used as a comparison group. In addition, it sought to determine whether it would be possible to construct a retrospective comparison group for fourth, fifth and sixth graders currently in the program.
Researchers worked with two schools for the study: Kelly and King. As part of the project, researchers:
- Consulted experts about appropriate measures and risk factors to study.
- Developed and piloted assessment tools.
- Developed a process for getting families to participate in a study.
- Began collecting data through interviews and surveys with teachers, parents and children.
Throughout the study, researchers consulted extensively with Public/Private Venture's Research Advisory Committee. (See the Appendix for a list of members.)
As part of the feasibility effort, Public/Private Ventures collected baseline data for an initial cohort for a full impact evaluation, should one go forward. It subcontracted with Northwest Professional Consortium of Portland, Ore. to conduct interviews with parents and children.
During the 20002001 school year, consortium staff interviewed 40 first-graders and their parents (24 in the mentoring program and 16 who had been eligible the program but were not selected). The following year, staff members did a follow-up interview with 20 of the mentees and 10 from the comparison group, as well as a second cohort of 41 first-graders (24 mentees, 17 comparison).
Additionally, they interviewed 48 fourth- to sixth-grade mentees and their parents, in 2001, with follow-up interviews of 43 in 2002. By the conclusion of the project in October 2002, Public/Private Ventures had analyzed only the first-year data for the initial cohort of first-graders.
The grantee reported the following findings to RWJF in 2002.
- Program youth exhibited a greater number of delinquent/negative behaviors than did comparison youth, primarily due to more program youth having "had more than a few sips of alcohol over the last year," "throwing rocks or bottles at people" and "doing something dangerous." However, in several other areas such as "family members running away," "family members losing a job," "siblings being suspended" and "family members seriously injured or dying" program and comparison youth scored similarly.
- Teachers ranked the program youth as more seriously "at risk" than the comparison youth. On a scale of one (less risky) to 10 (will definitely fail without the program), program youth averaged a score of 7.17, while the comparison averaged 5.97. According to the principal investigator, however, the comparison youth also exhibited behaviors that would make them appropriate for the program.
- There do not appear to be many significant or systematic differences between the program and comparison groups. There seems to be a fine line between those selected and not selected for the program. Reasons for non-selection include having siblings in the program, being involved in other youth programs, having problems (e.g., psychological or developmental) beyond the scope of the program, having a solid family structure and exhibiting few and minor behavioral problems.
- Researchers determined that creating a retrospective comparison group of fourth to sixth graders who had not made the final cut as first graders was impossible, since no documentation on them had been saved. The Research Advisory Committee felt strongly that, from a theoretical perspective, a retrospective comparison group would not be convincing to an outside audience. The committee suggested gathering and examining current data from the fourth- to sixth-graders in the program and comparing the behaviors and attitudes to national samples of youth. Additionally, it suggested examining the observational data and teacher rating information from when the youth were in first grade (which is available) to establish the progress the youth had made.
AFTER THE GRANT
Public/Private Ventures submitted proposals for both an impact evaluation and a study of the Friends program's replication to RWJF. RWJF turned both down, in part because of changes in its funding focus and in part out of concern that the proposed impact evaluation design was not a valid way to assess outcomes for the children.
The Foundation's community health focus was moving away from general social connectedness efforts like mentoring, toward more specific social supports. RWJF also believed that the quasi-experimental design of the impact evaluation (i.e., based on a comparison of two groups that scored differently in a risk assessment) would lack the validity of a comparison between randomly assigned groups.
For the purposes of an evaluation, the Public/Private Ventures researchers and Friends of the Children have since agreed to randomly assign eligible students from each school to program and comparison groups, in order to strengthen the validity of the evaluation. In fact, the program staff has come to view random assignment as a way to determine which children in the "gray area" of need are most likely to benefit from their services. Public/Private Ventures plans to renew efforts to secure funding for the evaluation.
GRANT DETAILS & CONTACT INFORMATION
Evaluating a Model Program for Mentoring High-Risk Children
Public/Private Ventures (Philadelphia, PA)
Dates: July 2000 to October 2002
Jean B. Grossman, Ph.D.
Public/Private Ventures Research Advisory Committee
Jacquelynne S. Eccles, Chair
Professor of Psychology
University of Michigan
Ann Arbor, Mich.
Lecturer in Public Policy
Kennedy School of Government
Professor of Economics
Professor of Economics and Public Affairs
Katherine S. Newman
Professor of Urban Studies
Kennedy School of Government
Professor of Demographic Studies and Sociology and Public Affairs
Report prepared by: Robert Crum
Reviewed by: Robert Narus
Reviewed by: Molly McKaughan
Program Officer: Laura C. Leviton