April 2007

Grant Results

SUMMARY

From 2000 through 2002, researchers at Social Capital Development Corporation analyzed outcome data from more than 200 school-based drug education/prevention programs.

The study employed updated meta-analytic research techniques to re-examine a 1999 analysis of the programs funded by the Robert Wood Johnson Foundation (RWJF).

Key Findings
In contrast to previous findings, the researchers found that:

  • Programs that teach comprehensive life skills — such as assertiveness, decision-making and goal setting — were not more effective than social influences programs-which focus on helping young people resist the pressures to use substances.

    These findings held across all grade levels and all substances examined.
  • There was no statistically significant difference, across all grade levels, in the effectiveness of interactive programs — which foster interpersonal skills and active engagement between students and teachers — and lecture-oriented, non-interactive programs, which stress drug knowledge.

Funding
RWJF supported the research with two grants totaling $299,712.

 See Grant Detail & Contact Information
 Back to the Table of Contents


THE PROBLEM

To use funds from the federal Department of Education's Drug-Free Schools Program, schools must implement evidence-based prevention programs. According to a 1999 study conducted by researchers at University of North Carolina, many schools use prevention programs that have not been proven to be effective.

To compare the effectiveness of diverse programs, researchers increasingly turn to meta-analysis, a technique that combines and analyzes data across many studies so that small effects can be detected. In 1999, researchers at Tobler Research Associates, a research firm working in the area of drug prevention, compiled over 200 studies, the largest data set of school-based drug education/prevention programs currently available.

Their initial meta-analysis, supported by RWJF (see Grant Results on ID# 034433), found that interactive programs, which foster interpersonal skills and active engagement between students and teachers, were more effective than lecture-oriented, noninteractive programs that stress drug knowledge.

The researchers also found that among the group of interactive programs, those that teach comprehensive life skills, such as assertiveness, decision-making and goal setting, were more effective than social influences programs, which focus on helping young people resist the direct and indirect pressures to use substances.

In a further finding, the 1999 study cast doubt on the effectiveness of the most popular prevention programs. Michael Roona, a senior research associate on that project who is now with Social Capital Development Corp, continued to refine the data after the death in February 2000 of Tobler Research's founder, Nancy Tobler. He is the principal investigator on the meta-analysis conducted under these two grants.

 Back to the Table of Contents


RWJF STRATEGY

In its work to reduce the harm caused by substance abuse, RWJF has supported a number of projects aimed at identifying effective school-based prevention programs. Based on findings from the 1999 Tobler study, starting in 2001, RWJF supported a series of grants to revamp and evaluate the largest and best-known drug prevention program, Drug Awareness Resistance Education (D.A.R.E.) (ID#s 037809, 039223, 040345, 040371).

The new D.A.R.E. curriculum targets 7th and 9th grade students, since research shows that students are most likely to be exposed to drugs as they enter middle school and high school. In this interactive program led by uniformed police officers, students practice in a group how to respond to social influences that encourage substance use. The new program was tested in 83 high schools and 122 middle schools. The evaluation found that seventh graders using the curriculum improved their decision-making skills, drug refusal skills and beliefs that drug use is socially inappropriate.

RWJF supported the project described here to further refine research to determine which kinds of interventions work best at different grade levels.

 Back to the Table of Contents


THE PROJECT

Researchers at Social Capital Development Corporation took the data set compiled by Tobler Research Associates and performed a number of new analyses. Under the first grant (see Grant Results on ID# 034433), they compared the effectiveness of two interactive prevention programs, one teaching comprehensive life skills, the other focused on social influences, to assess differences in effectiveness by grade level and substance used.

They also refined the outcome data to identify different levels of alcohol use — from students who consumed alcohol relatively frequently or in relatively large quantities to those whose consumption pattern was less frequent, in smaller quantities, or unknown.

Under the second grant (ID# 040928), researchers restructured the database, recomputed the size of the effect of both interventions and coded additional data so that it could be analyzed using new statistical techniques. They then tested Tobler's original finding that interactive programs were more effective than non-interactive ones, by looking at different age groups, substances used, and the length of time between the beginning of each program and the collection of post-test data used to assess program effectiveness.

The findings differed significantly from those of the 1999 study because of the differing methodological approaches used. Scientific knowledge is provisional and contingent upon research methods. Meta-analytic methods for drawing conclusions across a range of studies have continued to evolve in the social sciences.

In this instance, the currently accepted methodology is considered more precise than that used in the original Tobler meta-analysis; as a result, researchers were better able to identify the elements of drug prevention programs that truly affect outcomes. For a full description of the methodology employed, contact principal investigator Roona at mroona@social-capital.org.

 Back to the Table of Contents


FINDINGS

Researchers reported these findings from the first phase of their work at the RWJF-sponsored Prevention 2000 summit (ID# 039287) (see Communications):

  • A reanalysis of data from the 1999 Tobler study indicated that comprehensive life skills programs were not more effective than social influences programs, at all grade levels and with all types and levels of substance use.
    • Comprehensive life skills programs appeared to work better at the elementary and senior high school levels, but only in terms of an impact on students' use of cigarettes and hard drugs. There was no significant difference in impact on students' use of marijuana or alcohol.
    • At the middle school level, there was no difference between the effectiveness of social influences and comprehensive life skills programs for reducing the use of cigarettes, marijuana and other drugs. Social influences programs were moderately effective at reducing heavy drinking among middle school students, but ineffective at reducing alcohol use.

Researchers reported these findings from the second phase of their study in a poster session at the June 2003 meeting of the Society for Prevention Research (see Communications):

  • Using new modeling techniques, the overall difference between the effectiveness of interactive and non-interactive programs across all grade levels was found to be not statistically significant. Interactive programs were found to be significantly more effective than non-interactive programs at delaying, reducing, or preventing tobacco use, but not the use of other substances. When broken down by grade level, interactive programs worked better in middle school but not in high school.
  • The duration of a school prevention program may account for the large differences Tobler found between interactive and non-interactive programs. The beneficial effect of interactive programs occurs more than 18 months after pretesting, rather than a year, as Tobler reported.

Communications

The investigators prepared a background paper summarizing the first phase of their study, "Identifying Effective School-Based Substance Abuse Prevention Interventions," for the RWJF Prevention Summit (see Grant Results on ID# 039287) held October 30–31, 2000 in St. Michaels, Md. About a dozen top prevention researchers and 40 other professionals in the field attended this event, and used this paper and two others to help guide their discussion about ways to move effective programs into practice.

The investigators made several presentations about their methodology and findings, including at the annual meetings of the Drug Policy Foundation in May 2002 and a poster presentation at the Society for Prevention Research in June 2003. Findings from the meta-analysis also were included in two book chapters in the Encyclopedia of Primary Prevention and Health Promotion published in 2003 (see the Bibliography for details about communications products).

 Back to the Table of Contents


LESSONS LEARNED

The project produced two lessons for the field:

  1. Prevention researchers need to "vet" their work with other expert researchers before presenting results in the field that may influence public policy. (Project Director)
  2. Researchers need a forum where they can have in-depth discussions about methodology since different ways of analyzing the same data may yield very different, even contradictory, results. (Project Director)

 Back to the Table of Contents


AFTER THE GRANT

The research project concluded with this grant. Since the grants ended, the investigators have been preparing papers for publication in a peer-reviewed journal.

 Back to the Table of Contents


GRANT DETAILS & CONTACT INFORMATION

Project

Analyzing Data On Universal School-Based Drug Education Program Evaluations

Grantee

Social Capital Development Corporation (Albany,  NY)

  • Analyzing Data On Universal School-Based Drug Education Program Evaluations
    Amount: $ 50,000
    Dates: April 2000 to December 2000
    ID#:  039149

  • Assessing the Effectiveness of School-based Drug Education Programs
    Amount: $ 249,712
    Dates: February 2001 to January 2003
    ID#:  040928

Contact

Michael Roona
(518) 433-1755
mroona@social-capital.org

 Back to the Table of Contents


BIBLIOGRAPHY

(Current as of date of this report; as provided by grantee organization; not verified by RWJF; items not available from RWJF.)

Book Chapters

Roona MR, Streke AV and Marshall DG. "Substances, Adolescence (Meta-Analysis)." In Encyclopedia of Primary Prevention and Health Promotion, Gullotta TP & Bloom M (eds.). New York: Kluwer Academic/Plenum Publishers, 2003.

Streke AV, Roona MR, and Marshall DG. "Substances, Childhood (Meta-Analysis)." In Encyclopedia of Primary Prevention and Health Promotion, Gullotta TP & Bloom M (eds.). New York: Kluwer Academic/Plenum Publishers, 2003.

Articles

Roona MR and Eyle A. "Poor Results Prompt D.A.R.E. to Create New Curriculum: One Drug Education Expert Wonders… Are We Doing Enough?" The ReconsiDer Quarterly, 1(4): 2–5, 2002. (This article is available online.)

Reports

Roona MR, Streke AV, Ochshorn P, Marshall DM and Palmer AP. "Identifying Effective School-Based Substance Abuse Prevention Interventions," background paper for Prevention 2000 Summit. Albany, N.Y.: Social Capital Development Corporation, 2000.

 Back to the Table of Contents


Report prepared by: Kelsey Menehan
Reviewed by: Karyn Feiden
Reviewed by: Molly McKaughan
Program Officer: Katherine Kraft

Most Requested