Bravo! Your Idea Has Legs. Now Comes the Hard Part: Replicating It
Between 1993 and 1996, researchers at Cornell University, New York State College of Human Ecology, Ithaca, N.Y., evaluated the process by which successful demonstration projects are replicated into real-world programs.
They specifically examined the replication process within four Robert Wood Johnson Foundation (RWJF)-funded national programs and one project, as well as to recommend effective replication methods.
Transforming successful demonstration projects into real-world programs is a relatively new endeavor in the social service field. It was anticipated that the findings would be disseminated and would assist RWJF and other funding agencies in designing programs.
- The investigators identified two main replication strategies. These included:
- simple replication, in which sites receive training and technical assistance in adopting the methods of a well-tested program.
- developmental replication, in which a more flexible approach is taken for programs whose methods and outcomes have not been explored as well.
- Among the investigators' conclusions:
- Replication sites value one-to-one technical assistance more highly than any other kind of support.
- Grant funds appear critical in getting replication projects started, but once momentum has been achieved, the importance of the funds decreases.
- Organizations must buy into the project and provide the resources and personnel to implement and sustain it, instead of relying on grant funds.
- It may take up to a decade to recruit a critical number of sites that have adopted the replication project.
RWJF supported the project with a grant of $235,542 between August 1993 and April 1996.
Replication is the process of transferring an innovation to additional sites. In the business world, franchising and technology diffusion are common. In the fields of education, agriculture, and job training, the reproduction of successful pilot programs has been a practice since World War II.
In the delivery of health care and other human services, however, replication is a relatively new development. Although significant amounts of public and private funds have been spent finding new solutions to social problems, only a small number of the resulting innovations have been put to use beyond their initial demonstration sites.
This has begun to change in recent years, as constraints on social service spending have increased and led to a greater interest in maximizing the return from promising new programs.
During the early 1990s, RWJF program staff recognized that limited capital and regulatory obstacles made it difficult for new ideas, no matter how promising, to take off on their own, and they began developing programs to expand successful demonstration projects. Between July 1991 and December 1992, RWJF's Board of Trustees authorized nine of these replication efforts totaling approximately $78 million in grants and technical assistance.
With these new programs came an opportunity to study the replication process and to learn more about what it takes to transfer experience and lessons learned from a successful demonstration project to other sites.
This grant provided funding to review the replication field generally and to analyze RWJF-supported replication efforts, specifically. The objective was to provide information on the range of available replication strategies and to make recommendations on four issues:
- structuring demonstration projects to minimize the need for replication funding;
- developing the most productive replication strategies;
- understanding the circumstances associated with the successful diffusion of innovations;
- developing evaluation approaches that support replication.
Andrea Kabcenell, a former RWJF Program officer, was chosen to serve as principal investigator.
In addition to helping RWJF staff design future programs, the results were expected to be valuable to other public and private funding agencies. It was anticipated that findings would be disseminated through journal articles, monographs, a national meeting of funding agencies, and participation in groups interested in replication.
This was thought to be the first attempt to systematically assess diverse replication approaches, analyze their effectiveness as viewed by participants, funders, and policymakers, and identify early markers of replication success.
The project was planned in two phases:
- A review of the general field of replication through study of related literature and interviews with experts. The result was to be a detailed understanding of program replication.
- A close examination of four RWJF national programs engaged in replication initiatives at the national level and a replication project. This would involve visits to the national program offices overseeing the four programs, the grantee on the fifth project, and a survey by mail of the organizations carrying out the replications at the local level. The result was to be an in-depth understanding of the history, process, and early outcomes of RWJF replications.
Four multi-site national programs and one individual grant-funded project were chosen to reflect a mix of funding levels and replication strategies:
- Faith in Action®: The Interfaith Volunteer Caregivers Program, which supports the establishment of up to 1,100 interfaith volunteer caregiver projects for all ages with chronic health conditions.
- Join Together, which supports coordinated local efforts against substance abuse.
- Making the Grade: State and Local Partnerships to Establish School-Based Health Centers, which is a program to expand comprehensive health services for school-age children by funding school-based health centers that would be eligible for long-term support through state and local funding policies.
- Partners in Caregiving: The Dementia Services Program, which provides technical assistance and grant support to help adult day centers develop and strengthen innovative center-based, in-home, and other respite programs for people with chronic cognitive disorders.
- The Technical Assistance Collaborative (ID# 020138), which supported technical assistance, training, and dissemination based on earlier Foundation-funded programs for the mentally ill.
The project staff reviewed replication literature (which is largely in the fields of education, labor and job training, and agriculture), interviewed 13 experts in the field, made two site visits to each of the four national program offices and to the Technical Assistance Collaborative, and surveyed the local grantees and others participating in these efforts. A number of other replication programs, funded by RWJF and others, were reviewed in less detail.
The project staff produced two interim reports The Knowledge Base for Studying Replication and A Conceptual Model for Studying Replication which approached the subject of replication from a theoretical perspective. The project's final report Moving from Innovation to Institutionalization: An Evaluation of RWJF Replication Programs contains observations and conclusions about how the replication process works generally, and recommendations about replication strategies.
- The investigators identified two major approaches to replication: simple replication and developmental replication. For summaries of how each of the RWJF replication initiatives used one, or a combination, of these approaches, see the Appendix.
- Simple Replication is used when the innovative strategy has been tested and its methods and expected outcomes are well understood. In this case, it is a matter of describing the key elements of the strategy and methods and sharing them with people motivated to use them. Providing technical assistance and connecting credible experts with the new adopters are essential in this case.
- Developmental Replication is needed in circumstances where the innovative approach has not been well tested, when the outcomes are not reliable, or the field is changing rapidly. In this case, technical assistance is more intensive and must adapt to different circumstances. Frequently, increased funding and monitoring/evaluation of the participants is needed.
- National Program Offices that were working on simple replications saw their tasks as imparting key elements and convincing the participating organizations to adopt them. Credibility of the original program results and experts, along with sharing the materials that were most useful seemed to be the essence of their work.
- For National Program Offices that were developing a new approach to the original programs, the work was much more complex. They spent great effort imparting the key ideas of the original program to participants. However, just as much time was spent on understanding and influencing policymaking and developing an understanding of how to make innovation work in different circumstances.
- Though the grantees and participants in these five replication initiatives were quite varied with respect to the resources and support they used, they were in agreement about the value of the resources and support received:
- One-to-one technical assistance was seen as the most useful support offered by the program.
- Attendance at national meetings was the second most valuable support, in part, because it offered the opportunity for collaboration with peers.
- Access to templates and other materials were the next most important support the participants received. Newsletters, literature, and other information materials were useful for raising awareness but not seen as central to participants' efforts.
- For those organizations that received grants, the funding seemed important to starting the replication, but was no longer critical once they had begun to gather some momentum. For those who did not receive grant money, 5 percent said that lack of funds was hampering their efforts.
- Some factors are key. These were gathered from site visits, participant surveys, and interviews with key informants:
- Replication requires perseverance. In many cases up to 10 years will be required to reach a critical number of adopters.
- Replication requires credibility. The leaders must have evidence that their innovation works and provide adopters with access to people running projects that have been successful.
- Replication works best if the field is not shifting. If the environment is unsteady, the work to disseminate is much more complex, and the "copying" that speeds replication is less useful. Instead, messages must be tailored to each circumstance.
- Developmental replications must be much more flexible and adaptive. Often, developmental replications are not sufficient to institutionalize the innovation, and further efforts must follow.
- Too much money has been devoted to replication. Building good will and offering a credible innovation with strong technical assistance is more effective (but harder work) than giving grant funds.
- Organizational buy in is essential, in the form of senior leadership commitment of staff and funds. If grant funds are used in replication, they should be front-loaded to assist participants with planning. The resources to implement the intervention in participating organizations should come from the organizations themselves, and not from grant funds.
- Technical assistance is by far the most effective strategy. The more connection between participants and credible experts, the more quickly dissemination will take place. Particularly during the planning phase, access to templates, materials and someone to talk to will have the greatest impact.
The project did not produce an in-depth analysis of the effectiveness of the four RWJF national programs and one replication project as originally planned, because these initiatives were still underway during the grant period, making it difficult to draw conclusions regarding the effectiveness of the replication efforts. However, the principal investigator was able to conclude that RWJF's replication programs had sufficient flexibility in design so that they could adapt to the changing environments and conditions. Several had increased both the technical assistance they provided and the policy work and research they planned.
A project staff member made presentations on the project at an American Evaluation Association international meeting in Vancouver, B.C., in November 1995, and at two national conferences on evaluation. There was no other dissemination of project findings.
AFTER THE GRANT
The principal investigator has had informal discussions with the program officer about extending the evaluation once the five programs had closed. No post-grant activities for this project are planned at this time.
GRANT DETAILS & CONTACT INFORMATION
Evaluation of the Robert Wood Johnson Foundation's Replication Programs
Cornell University New York State College of Human Ecology (Ithaca, NY)
Dates: August 1993 to April 1996
Andrea I. Kabcenell, R.N., M.P.H.
(607) 255-0171, (607) 539-3229
RWJF Replication Initiatives Studied
Each of the five initiatives used a different combination of dissemination strategies to increase the number of organizations using the program intervention.
- Partners in Caregiving: The Dementia Services Program, provides technical assistance and grant support to help adult day centers develop and strengthen innovative center-based, I-home, and other respite programs for people with chronic cognitive disorders. Projects were either given a grant of up to $300,000 plus technical assistance (TA), or they received technical assistance only. The TA included one-to-one contact with experts; training in marketing, service development financial management and communications; templates of important materials for the organizations to adapt; participation in meetings to allow for collaboration; and access to information such as newsletters, and scholarly literature.
- Faith in Action: The Interfaith Volunteer Caregivers Program, supports the establishment of up to 1,100 interfaith volunteer caregiver projects for all ages with chronic health conditions. This program gave grants of $50,000 to groups of religious organizations in a community to create supportive networks of health services. In addition to funding, replication strategies included providing organizations with standard materials to adapt, one-to-one access to expert mentors, attendance at national meetings, and information in the form of newsletters and other literature.
- Making the Grade: State and Local Partnerships to Establish School-Based Health Centers, is a program to expand comprehensive health services for school-age children by funding school-based health centers that would be eligible for long-term support through state and local funding policies. Grants were made to individual states to foster the growth of school-based health services for children and adolescents. Replication strategies included technical assistance from national experts, attendance at national meetings, access to materials and literature, and considerable policy work from the national program office, which researched and produced "white papers" and other materials for legislators and other policy makers. The focus on states rather than health care providers was a departure from the initial program, and thus required considerable developmental work.
- Technical Assistance Collaborative, supported technical assistance, training, and dissemination based on earlier RWJF-funded programs for the mentally ill. This project largely supported technical assistance provided by the deputy director of the original RWJF program for the mentally ill. With his staff, the principal investigator worked with counties, cities, and some states to develop community based services for people with serious mental illnesses. As many states shifted health care financing to capitation, a system that pays health plans a fixed fee per person to provide all needed care, the work shifted to teaching cities and states how to build capitated mental health services. Strategies included publishing policy and position papers; access to materials and literature, and site visits from the staff who worked to facilitate meetings and train participants. Because the field was changing, the strategies taught had to be developed and modified for different environments.
A Combination Approach
- Join Together, an organization at Boston University School of Public Health, supports coordinated local efforts against substance abuse. RWJF started funding the work of Join Together during the second year of its separate Fighting Back® Program, a national community substance abuse prevention program, for which Join Together has now become the national program office. In addition to disseminating the program model developed by Fighting Back®, Join Together also:
- trained fellows to act as future leaders in the area;
- ran a Web site and listserv information service on service strategies and funding sources;
- consulted with communities to bring relevant parties together;
- held yearly national meetings to disseminate key strategies;
- published multiple newsletters, position papers, and other documents to increase awareness and knowledge among the public and health care providers.
(Current as of date of this report; as provided by grantee organization; not verified by RWJF; items not available from RWJF.)
Books and Reports
Chang L, Geisz M, Grazier K, Greene J and Kabcenell A. The Knowledge Base for Studying Replication: An Interim Report of the Evaluation of RWJF's Replication Programs. Ithaca, N.Y.: Cornell University, 1994.
Geisz M, Grazier K, Greene J and Kabcenell A. A Conceptual Model for Studying Replication: Interim Report to the Robert Wood Johnson Foundation. Ithaca, N.Y.: Cornell University, 1995.
Geisz M, Grazier K, Greene J and Kabcenell A. Moving from Innovation to Institutionalization: An Evaluation of RWJF Replication Programs. Ithaca, N.Y.: Cornell University, 1996.
Presentations and Testimony
Mary Geisz, "The Evaluation of Social Program Replication," at the Edward F. Kelly Evaluation Conference, Cornell University, Syracuse University and State University of New York at Albany, March 18, 1994, Albany, N.Y.
Mary Geisz, "Evaluation of Program Replication: Frameworks, Models and Strategies," at the Edward F. Kelly Evaluation Conference, Cornell University, Syracuse University, and State University of New York at Albany, March 31, 1995, Syracuse, N.Y.
Mary Geisz, "Evaluation of Program Replication: Funding Strategies That Work," at Evaluation '95: International Evaluation Conference, American Evaluation Association, November 1995, Vancouver, B.C.
Report prepared by: Michael H. Brown
Report prepared by: Richard Camer
Reviewed by: Molly McKaughan
Reviewed by: Marian Bass
Program Officer: Joel C. Cantor
Program Officer: Beth A. Stevens