August 2001

Grant Results

SUMMARY

From 1997 to 1998, the Community Anti-Drug Coalitions of America, Alexandria, Va., oversaw the development and preliminary assessment of a computer-based interactive, multimedia software program designed to help prevent alcohol and marijuana use among 11- to 15-year-olds.

The World Institute of Leadership and Learning, a for-profit, women-owned, small business specializing in multimedia and video technologies, health communications, and the design and development of instructional systems, Potomac, Md., produced the game.

The company earlier developed a specialized technology known as the Komputer Interactive Simulation System® (KISS®) technology.

Key Results

  • The World Institute of Leadership and Learning applied its trademarked KISS computer technology to develop "Substance Abuse Interactive Nights Out." KISS uses a variety of interactive media to allow players to access information, role-play, experiment with "what if " situations, and experience first-hand the consequences of their choices, all in the safety of a virtual environment. Scenarios are incorporated into video games, portrayed by actors who talk to teens and young adults in their own language.
  • "Substance Abuse Interactive Nights Out" included:
    • Two interactive movie video games, one for males and one for females.
    • Two quizzes about substances and related topics.
    • A keyword search (encyclopedia) of topics related to substance abuse.
    • Screens of substance abuse prevention resources.
    • User response tracking software.
    • A jukebox with popular music.
  • Students from Suitland High School in Maryland reviewed the program regularly during development.
  • Staff attended several conferences and met with educators, substance abuse and public health professionals, and youth to research program structure and content and to obtain feedback.
  • A small, preliminary assessment conducted by researchers at the Boston University School of Health indicated students, parents, and teachers believed the program to be appropriate for seventh and eighth graders.

Funding
The Robert Wood Johnson Foundation (RWJF) supported the project with a grant of $499,716 between July 1997 and July 1998.

 See Grant Detail & Contact Information
 Back to the Table of Contents


THE PROBLEM

The World Institute of Leadership and Learning (WILL), a for-profit, women-owned small business specializing in multimedia and video technologies, health communications, and design and development of instructional systems, developed a specialized technology known as the Komputer Interactive Simulation System® (KISS®) technology. KISS uses a variety of interactive media to allow players to access information, role-play, experiment with "what if " situations, and experience first-hand the consequences of their choices, all in the safety of a virtual environment. Scenarios are incorporated into video games, portrayed by actors who talk to teens and young adults in their own language.

The first KISS project developed by WILL was "HIV/AIDS Interactive Nights Out," which targeted hard-to-reach populations, in particular, youth — a priority for The Robert Wood Johnson Foundation's (RWJF) substance abuse prevention efforts. After viewing that game, the Community Anti-Drug Coalitions of America (CADCA), a Foundation grantee supporting community coalitions against substance abuse throughout the United States, referred WILL to the Foundation.

RWJF asked WILL to review the literature on interactive technologies for changing substance abuse knowledge, attitudes, and behaviors to better ascertain the game's potential effectiveness and to improve its design. WILL reported finding nothing comparable in the literature to the product being proposed. However, WILL did identify studies that showed that (1) higher levels of interactivity lead to higher learning, and (2) interactive videodisc instruction was less costly than more conventional instruction.

 Back to the Table of Contents


THE PROJECT

RWJF's grant to support this project went to CADCA with the expectations that CADCA would subcontract with WILL and provide supervision and financial oversight to the project. CADCA took a less directive role than the Foundation had anticipated. CADCA's involvement did, however, add credibility to the effort, especially in the Cincinnati area where the product was tested on teenagers and CADCA was building a drug coalition. CADCA was able to open doors to WILL.

 Back to the Table of Contents


RESULTS

The Product

WILL applied its trademarked KISS computer technology to develop "Substance Abuse Interactive Nights Out," which included two interactive movie video games, one for males and one for females; two quizzes about substances and related topics; a keyword search (encyclopedia) of topics related to substance abuse; screens of substance abuse prevention resources; user response tracking software; and a jukebox with popular music. WILL intended to comply with the basic elements of school-based prevention programs summarized in Preventing Drug Use Among Children and Adolescents: A Research-Based Guide, published by the National Institute for Drugs and Alcohol, which conducts research and develops public policy.

  • Conducted focus groups with 11 to 15 year-olds from urban, suburban, and rural areas of California and Ohio. Students from Suitland High School in Maryland reviewed the program regularly during development.
  • Attended several conferences and met with educators, substance abuse and public health professionals, and youth to research program structure and content and to obtain feedback.
  • Designed and developed the instructional system using the KISS technology. A panel of experts from the fields of public health, substance abuse, education, and computer technology reviewed scripts prior to production. (See the Appendix for Program Review Board members.)
  • Completed casting, setting, filming, video editing, and computer programming.
  • Continued review of the completed program with the Program Review Board, youth, and other advisors. These reviews resulted in some restructuring of the game and suggested the need for future improvements, including voiceovers for low-literacy users, a Spanish and a Web version of the game, and a teacher's manual.

 Back to the Table of Contents


FINDINGS

The Pilot Evaluation

A team of researchers at Boston University School of Public Health, who were experienced in technology- based behavior modification interventions, conducted a small preliminary assessment.The objectives were to:

  • Collect quantitative information about the effect of viewing the game in terms of changing student knowledge about substance abuse, attitudes towards alcohol and marijuana, and intended behaviors with respect to future alcohol and drug use.
  • Collect qualitative data on reactions to the video by parents, teachers, and seventh and eighth graders who viewed the game.

The purpose of this pilot evaluation was not to definitively test the effectiveness of the video, but was intended to determine whether a larger evaluation of the product could be successfully implemented in a comprehensive evaluation study.

Using a quasi-experimental design, researchers studied the effects of viewing the video in one urban and one suburban junior high school in the Cincinnati area. More than 200 students in the seventh and eighth grade who had seen the video were compared to students in four control schools (two urban and two suburban) who had not. The study was conducted on personal computers installed in the libraries of the intervention schools.

The researchers gathered quantitative data on changes in student knowledge, attitudes, and intended behaviors from questionnaires administered before and after students played the game. Questionnaires asked students about such things as their alcohol and marijuana use in the last month; whether it was hard to refuse alcohol, marijuana, or other drugs if offered by a friend; and whether students planned to use these substances within the next month.

The researchers also gathered data on student perceptions of the video, as well as those of teachers and parents, since the latter are often gatekeepers to the substance abuse interventions their children can access. A self-selected group of 20 parents and 20 teachers were asked their opinions on whether the video was appropriate for seventh and eighth graders; whether it taught new information; whether it would change attitudes or future intentions, or delay or prevent future substance use; and whether it should be available to all seventh and eighth grade students.

The researchers also surveyed the students about whether they liked or disliked the video, whether they could relate to the characters, whether they learned anything, and whether they wanted to watch it again.

Key Lessons from the Pilot Evaluation

In the pilot assessment of Substance Abuse Interactive Nights Out, the researchers looked at the direction of the change in knowledge, attitudes, and behavior, rather than statistical significance, because the number of students studied was too small to detect statistical differences. Overall, the researchers concluded:

  • A larger-scale, school-based evaluation of the video could be successfully implemented.
  • Students, parents, and teachers believed the intervention to be appropriate and potentially effective.
  • Data yielded by the pilot study suggest that the video can be effective in changing substance use and substance use intentions among the targeted age groups.
  • There was sufficient evidence on potential effectiveness to warrant a large-scale efficacy trial.

Other key findings:

  • Most parents and teachers believed the video would be effective in teaching new information, changing attitudes, changing future intentions, and delaying onset of substance use.
  • Half the parents, but only 35 percent of teachers, thought the video would be effective in preventing future use among children already using substances.
  • Slightly fewer than three-quarters of the students reported they could relate to the characters.
  • Almost all urban students learned something from the game; in contrast, only 62 percent of suburban students reported learning something.
  • 94 percent of the students wanted to watch the games again.

Limitations

Evaluation staff at the Foundation voiced concern that this pilot evaluation study did not address certain important issues raised during the proposal development process. For example, the study did not address how frequency of viewing the video affected outcomes (the dose-response relationship.)

Furthermore, the study did not assess the game in the context of a broader school or community prevention effort, despite earlier research suggesting that single-modality interventions are ineffective. Evaluation staff also believe more definitive information is necessary about the game's ability to strengthen social skills, especially refusal skills, and to alter perceptions of substance use prevalence and social approval for use; perceptions of harm and risk; and perceived ability to resist use. Research has shown these factors to be critical in influencing youth substance use.

Communications

Because this grant was designed only to develop and assess a substance abuse prevention intervention, dissemination was not a project objective. However, both CADCA and WILL, believe the preliminary assessment provided sufficient evidence of effectiveness to begin further dissemination. At its own expense, in January 1999, WILL distributed 20,000 free CD-ROM sets (two games each) to Scholastic, Inc., a publisher of children's books available only through school systems. Scholastic, in turn, mailed the sets to middle-school teachers who subscribe to Choices, its life skills magazine, and Action, its publication for low-proficiency readers. These demo copies were programmed to go blank after several weeks, after which teachers had the option of buying the CD-ROMs for $300 per set. WILL also engaged a telemarketing firm to market the product to middle and junior high schools and attended education conferences to promote the product.

 Back to the Table of Contents


AFTER THE GRANT

The Foundation is unwilling to support further dissemination of the product without a full-scale evaluation of its effectiveness. Foundation staff referred WILL to the National Institute on Drug Abuse (NIDA) to pursue the possibility of funding for a larger-scale outcome study to assess the game's effectiveness. WILL reported that NIDA might be willing to support such a study, but not the wider product distribution that would be necessary first. The Foundation plans no further support for this activity.

 Back to the Table of Contents


GRANT DETAILS & CONTACT INFORMATION

Project

Development and Assessment of an Interactive Video to Prevent Substance Abuse Among Youth

Grantee

Community Anti-Drug Coalitions of America (Alexandria,  VA)

  • Amount: $ 499,716
    Dates: July 1997 to July 1998
    ID#:  031397

Contact

Nelson Cooney
(703) 706-0560

 Back to the Table of Contents


APPENDICES


Appendix 1

(Current as of the time of the grant; provided by the grantee organization; not verified by RWJF.)

Program Review Board Members

Naomi Abramowitz
Seventeen Magazine
New York, N.Y.

Gilbert J. Botvin, M.D.
Director
Institute for Prevention Research, Cornell University Medical College
New York, N.Y.

Helen Cain Jackson, M.S.W.
Administrator for Prevention Initiatives
Ohio Department of Mental Health
Columbus, Ohio

William DeJong, Ph.D.
Professor, Department of Health and Social Behavior
Harvard School of Public Health
Cambridge, Mass.

Jackie Gentry, Ph.D.
Director of Issues in the Public Interest
American Psychological Association
Washington, D.C.

Terry Gock, Ph.D.
Director, Asian Pacific Family Center
Staff Psychologist, Ingleside Hospital
Alhambra, Calif.

Rose Strickland
National Capital Coalition to Prevent Underage Drinking
Washington, D.C.

Jim Theophelis, M.C., Q.C.D.C.
Director of Clinical Programs, YouthCare
Seattle, Wash.

William Zeff
Former US Representative
Washington, D.C.

 Back to the Table of Contents


BIBLIOGRAPHY

(Current as of date of this report; as provided by grantee organization; not verified by RWJF; items not available from RWJF.)

Press Kits and News Releases

News release announcing selection of product for inclusion in President's launch of Youth Anti-Drug Media Campaign was emailed on July 2, 1998 to 40 newspapers, magazines, and television stations.

Audio-Visuals and Computer Software

Substance Abuse Interactive Nights Out. Interactive youth substance abuse prevention program. CD-ROM produced June 30, 1998 by World Institute of Leadership and Learning, Potomac, Md. 20,000 copies distributed, 50 copies sold.

 Back to the Table of Contents


Report prepared by: Janet Heroux
Reviewed by: Karyn Feiden
Reviewed by: Marian Bass
Program Officer: Floyd K. Morris