August 2001

Grant Results

SUMMARY

Starting in January 1994, the Johns Hopkins University School of Medicine conducted a three-year evaluation of Hawaii's Healthy Start Program (HSP).

HSP provides early intervention family support services to young children from birth to age 5 who are at risk for maltreatment. This evaluation sought to determine:

  • How closely the HSP implementation mirrored its design.
  • How effectively the program achieved its intended benefits for children and families.
  • How the fidelity of program implementation influenced program impact.
  • And how program benefits compared with program costs.

Key Findings

  • Early identification staff determined risk status for 84 percent of target families.
  • Families with higher risk scores, young mothers with limited schooling and families with infants at biologic risk were more likely to enroll in home visiting.
  • Only half of those who enrolled were active at one year with an average of 22 visits.
  • After two years of service provision to families, HSP was successful in linking families with pediatric medical care.
  • Most families enrolled for home visits were linked with a primary care provider, but linkage rates for other community resources varied widely by type of service.
  • Although HSP visits improved family wellness according to several measures, no overall positive program impact emerged after two years in terms of child maltreatment, adequacy of well-child health care, or pediatric health care use for illness or injury.

Conclusions

  • The researchers conclude in The Future of Children that "home visiting programs and evaluations should monitor program implementation for faithfulness to the program model, and should employ comparison groups to determine program impact."

Funding
The Robert Wood Johnson Foundation (RWJF) provided partial funding for this study with a grant of $400,000 from January 1994 to July 1999. The National Institutes of Health has committed more than $5 million in funding for the next phase of the study.

 See Grant Detail & Contact Information
 Back to the Table of Contents


THE PROBLEM

Hawaii's HSP is a nationally known home visiting program providing early intervention family support services to young children from birth to age 5 who are at risk for maltreatment. In 1994, it was the only mature, state-run home visiting effort in which the goal was to identify and prevent adverse health, social, and developmental outcomes in the families of all at-risk newborns.

Under the supervision of health or social services professionals, paraprofessional home visitors work with families to prevent child abuse and neglect, improve family functioning, improve child development, promote access to primary health services, maximize child health status, and promote readiness for school. From its founding in 1985, HSP had, by 1995, expanded to 14 sites statewide.

Findings from a 1990 evaluation strongly suggested that HSP successfully identified families at-risk for child abuse and neglect and averted these behaviors, although a weak design limited the value of this study. The results of this and other studies increased the interest of child health and child welfare advocates and some government officials in early intervention approaches centered on home visiting, as in the HSP model.

At the same time, a review of home visitation programs nationwide showed variation in the programs' objectives, target populations, structure, visitor qualifications and their corresponding effectiveness. Limited knowledge in general about the effectiveness of community-based, home visitation programs impeded efforts to disseminate the HSP findings.

 Back to the Table of Contents


THE PROJECT

This study sought to determine:

  1. How closely Hawaii's HSP program implementation mirrored program design.
  2. How effectively the program achieved its intended benefits for children and families.
  3. How the fidelity of program implementation influenced program impact.
  4. How program benefits compared with both direct and indirect program costs, including costs of other government programs it affected.

The study was designed to enable investigators to address issues unresolved in earlier evaluations of home visiting programs.

From November 1994 to December 1995, researchers randomly assigned 684 families identified by HSP staff as at risk for abuse or neglect of their newborn infants to either the treatment group (HSP group) or the control group, then followed them from 1995 through 1998. The pool of families from which the sample was derived provided informed consent for this screening ("early identification") at the time of hospital registration.

Families in the HSP group and the main control groups were tested annually through structured parent interviews, observation, and developmental testing of the child. Besides primary data gathered from the participating at-risk families, researchers used existing data from HSP, physician, insurance, and Hawaii's Child Protective Services records.

Early on in the study, researchers discovered so much variation in HSP program delivery from one agency to another that a full evaluation of the program's effectiveness was not possible until researchers could complete a study of the effects within each agency. Thus project staff fully addressed only the first of the four objectives—program implementation compared to program design—and partially completed the second and third. Work on the fourth objective—the cost-benefit analysis—was postponed because evaluators would not have access to the state's Medicaid claims data until 2000.

Other Funding

Funds from RWJF supported approximately 20 percent of this study. The balance plus additional time for data analysis came from the federal Maternal and Child Health Bureau ($836,932), the David and Lucile Packard Foundation ($667,595), the Annie E. Casey Foundation ($405,000), and the Hawaii State Health Department ($294,738).

An evaluation Advisory Committee was established to provide additional support and direction for the project. (See the Appendix.) The grantee also used some of the grant funds to pay a subcontractor—Dr. Barbara Wasik, professor and associate dean of the School of Education at the University of North Carolina, Chapel Hill—for technical assistance in the development of a measure for parent problem-solving abilities.

 Back to the Table of Contents


FINDINGS

The following findings come from the abstract of an article in Pediatrics (January 2000):

  • "Early identification staff determined risk status for 84 percent of target families." Risk status is the risk for abuse or neglect of newborn infants assessed, with family approval, using a screen of maternal medical records and a maternal interview based on a checklist of family stress—Kempe's Family Stress Checklist. Target families were all families experiencing births at the study's six community hospitals during the data-gathering period.
  • "Families with higher risk scores, young mothers with limited schooling, and families with infants at biologic risk were more likely to enroll in home visiting." Indicators of biologic risk were birth weight, gestational age, and intermediate/intensive neonatal care.
  • "Half of those who enrolled were active at 1 year with an average of 22 visits. Families where the father had multiple risk factors and where the mother was substance abusing were more likely to have 12 or more visits; mothers who were unilaterally violent toward the father were less likely." The HSP protocol for frequency of visits was based on an assessment of family need and ranged from weekly to quarterly visits.
  • "Most [enrolled] families were linked with a medical home"—a primary care provider.
  • "Linkage rates for other community resources varied widely by type of service."
  • "Half of [enrolled] families overall, but 80 percent or more of those active at 1 year, received core home visiting services"—the writing of an individualized family service plan, an infant developmental screening, and an assessment of mother-child interaction.
  • "Performance varied by program site." Performance refers to the number of visits and the percent of families who received core services. There were six program sites, each serving a geographically defined community; three community agencies administering HSP, each agency operating at two sites.

The following findings come from an article in The Future of Children (9(1): 66–90, 1999), based on findings from years 1 and 2 of this three-year study:

  • "After two years of service provision to families, HSP was successful in linking families with pediatric medical care, improving maternal parenting efficacy, decreasing maternal parenting stress, promoting the use of nonviolent discipline [of children], and decreasing injuries [to children] resulting from partner [biologic parent and/or parent partner] violence in the home." (These findings were derived from parent interviews and in-home assessments.)
  • "No overall positive program impact emerged after two years of service in terms of the adequacy of well-child health care; maternal life skills, mental health, social support, or substance use; child development; the child's home learning environment or parent-child interaction; pediatric health care use for illness or injury; or child maltreatment (according to maternal reports and child protection services reports)."
  • "There were agency-specific positive program effects [that is, individual effects that were specific to one community agency carrying out HSP, and not to another] on several outcomes, including parent-child interaction, child development, maternal confidence in adult relationships, and partner violence."
  • "Significant differences were found in program implementation between the three administering agencies included in the evaluation. These differences had implications for family participation and involvement levels and, possibly, for outcomes achieved."
  • "The authors conclude that home visiting programs and evaluations should monitor program implementation for faithfulness to the program model, and should employ comparison groups to determine program impact."

Limitations

This study used Hawaii's Child Protective Services (CPS) reports of child abuse as one measure of program impact. However, as the researchers point out in their article in The Future of Children, CPS reports "suffer at least two important limitations as a measure of program impact: (1) because reporting is a rare event, extremely large sample sizes are needed to identify meaningful reductions in reporting rates, with smaller sample sizes lacking the power to detect small, yet meaningful, reductions; and (2) there is a strong potential for reporting bias, which can operate either to inflate or deflate estimates of abuse and neglect."

Communications

Because the initial findings of this study were mixed and potentially damaging to home visiting programs in operation, the Packard Foundation released evaluation results gradually, first to a select group of practitioners and researchers. Wider dissemination began at a March 1999 National Academy of Science conference (sponsored by the Academy's Institute of Medicine and National Research Council) on early intervention for families at risk, at which the principal investigator gave a presentation on this study.

These events also coincided with the 1999 publication of study results in the Spring/Summer 1999 issue of the Packard Foundation's journal The Future of Children, which was devoted to recent evaluations of home visiting programs. Results have since been published in Pediatrics. Investigators also have made 28 presentations of their findings to national and international meetings. See the Bibliography for more details.

 Back to the Table of Contents


LESSONS LEARNED

  1. Evaluative research is likely to be most effective when it communicates timely feedback for program development. The Hawaii Department of Health, responding to early evaluation reports of departure from the HSP model, convened a statewide quality improvement planning group of HSP directors and supervisors. This group used evaluation findings to examine program implementation and attrition, learn from the successful practices, reassess program standards, and recommend practices to improve service quality.
  2. Even with established home visiting programs, service evaluations: (1) must measure delivery of services and (2) establish a clear "denominator" to report service delivery. Delivery of services often has not been measured in evaluations of home visiting programs. In this study, the agencies involved, after several years' administration, did not know that their rates of attrition varied from 38 to 64 percent, and that the administering agencies' philosophies affected the rate of attrition from a home visiting program. Another factor affecting evaluation results is the denominator used to measure program delivery. If evaluations look at services delivered to "all the families originally enrolled in the program," family attrition from the program can dilute the strength of program process that is reported. If the denominator for evaluating service delivery is "families that are active in the program over the period of the evaluation," however, the program delivery reported will look quite different.
  3. The ability to generalize conclusions based on program evaluations using a single agency is limited. Even though home visitor training and contract stipulations were the same for all agencies, some HSP agencies were able to promote some aspects of family functioning and child development while others were not. This makes it problematic to generalize conclusions from findings on clients at one agency.
  4. While program quality is essential, linkages among programs also need evaluation. HSP was intended to provide service directly as well as to link families with other resources. Researchers need to look at how home visiting programs relate to other resources in the community. According to the project director, more research is needed on the impact of service integration on child and family outcomes. More sophisticated measures of the linkages among these services are also needed.

 Back to the Table of Contents


AFTER THE GRANT

Analysis of how HSP program implementation affects outcomes and a cost-benefit analysis of the HSP program continue with additional support from the federal Maternal and Child Health Bureau, the Hawaii Department of Health, and the Packard Foundation.

The National Institutes of Health has committed more than $5 million in funding for the next phase of the study, which will assess the effectiveness of early home visiting in promoting children's mental health, development, and academic achievement by following HSP children from kindergarten through third grade.

 Back to the Table of Contents


GRANT DETAILS & CONTACT INFORMATION

Project

Evaluation of Hawaii's Healthy Start Program

Grantee

The Johns Hopkins University School of Medicine (Baltimore,  MD)

  • Amount: $ 400,000
    Dates: January 1994 to July 1999
    ID#:  018303

Contact

Anne K. Duggan, Sc.D.
(410) 614-5280
aduggan@jhu.edu

 Back to the Table of Contents


APPENDICES


Appendix 1

(Current as of the time of the grant; provided by the grantee organization; not verified by RWJF.)

Hawaii's Healthy Start Program Evaluation Advisory Committee

Gigliola Baruffi, M.D., M.P.H.
Associate Professor and Head
Maternal and Child Health Program
School of Public Health
University of Hawaii at Manoa
Honolulu, Hawaii

Gail Breakey, R.N., M.P.H.
Director
Hawaii Family Stress Center
Honolulu, Hawaii

Stephanie Feeney
Professor of Education
College of Education
University of Hawaii at Manoa
Honolulu, Hawaii

Loretta J. Fuddy, A.C.S.W., M.P.H.
Acting Program Administrator
Maternal and Child Health Branch
State of Hawaii Department of Health
Honolulu, Hawaii

Setsu Furuno, Ph.D.
Chairperson
Hawaii Early Intervention Coordinating Council
Honolulu, Hawaii

S. L. Hammar, M.D.
Professor and Chairman
John A. Burns School of Medicine
University of Hawaii at Manoa
Honolulu, Hawaii

Valma A. Kameoka, Ph.D.
Director
Social Welfare Evaluation, Research, and Training Unit
School of Social Work
University of Hawaii at Manoa
Honolulu, Hawaii

Jean L. Johnson, M.A., M.P.H.
Project Coordinator
Zero-to-Three Hawaii Project
State of Hawaii Department of Health
Honolulu, Hawaii

Joseph P. O'Reilly, Ph.D.
Associate Professor of Human Resources
College of Tropical Agriculture and Human Resources
University of Hawaii at Manoa
Honolulu, Hawaii

Dexter S. Y. Seto, M.D.
Professor of Pediatrics
University of Hawaii
Honolulu, Hawaii

Sharon Taba
Associate Director
The Medical Home Project
Hawaii Medical Association
Honolulu, Hawaii

Rob Welch, Ph.D.
President and CEO
Child and Family Services
Honolulu, Hawaii

Ed Yoshimoto
Services Program Development Administrator
State of Hawaii Department of Human Services
Honolulu, Hawaii

 Back to the Table of Contents


BIBLIOGRAPHY

(Current as of date of this report; as provided by grantee organization; not verified by RWJF; items not available from RWJF.)

Articles

Buchbinder SB, et al. "Home Visitor Job Satisfaction and Turnover." The 10th Annual Research Conference Proceedings, A System for Care for Children's Mental Health, Expanding the Research Base. C Liberton, K Kutash and R Friedman (eds). Tampa, FL: University of South Florida Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children's Mental Health, 1997.

Buchbinder SB, et al. "The Impact of Home Visitor Turnover on Remaining Co-Workers." The 10th Annual Research Conference Proceedings, A System for Care for Children's Mental Health, Expanding the Research Base. C Liberton, K Kutash and R Friedman (eds). Tampa, FL: University of South Florida Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children's Mental Health, 1997.

Duggan AK, McFarlane EC, Windham AM, Rohde CA, Salkever DS, Fuddy L, Rosenberg LA, Buchbinder SB and Sia CCJ. "Evaluation of Hawaii's Healthy Start Program." The Future of Children: Home Visiting, Recent Program Evaluations, 9(1): 66–90, 1999. Also appears online.

Duggan A, Windham A, McFarlane E, Fuddy L, Rohde C, Buchbinder S and Sia C. "Hawaii's Healthy Start Program of Home Visiting for At-Risk Families: Evaluation of Family Identification, Family Engagement, and Service Delivery." Pediatrics, 105(1): 250–259, 2000. Also appears online.

Windham AM, et al. "Comprehensive Evaluation of the Hawaii Healthy Start Program: Effects on Maternal Mental Health, Substance Use and Social Support at One Year." In The 11th Annual Research Conference Proceedings, A System for Care for Children's Mental Health, Expanding the Research Base edited by C Liberton, K Kutash and R Friedman. Tampa, FL: University of South Florida Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children's Mental Health, 1998.

Presentations and Testimony

Anne K. Duggan, et al., "Home Visiting Programs: Criteria for Evaluative Research Design," at Changing Directions: Strategies for Working with At-Risk Families, California Department of Social Services, June 1995, Sacramento, CA.

Anne K. Duggan, et al., "Evaluation of Hawaii's Healthy Start Program," an invited workshop presentation at Healthy Mothers, Healthy Babies Coalition 1995 National Biennial Conference, October 1995, Arlington, VA.

Sharon B. Buchbinder and Anne K. Duggan, "Evaluation of the Hawaii Healthy Start Program," at the Ambulatory Pediatric Association Region IV Annual Meeting, January 1996, Richmond, VA.

Anne K. Duggan, et al., "Examining What We Know About Home Visiting and Construction of a Research Agenda for the Future," at Within Our Reach, Effective Home-Based Strategies for Family Support, sponsored by the California Department of Social Services, the California Department of Health, the Stuart Foundation, the California Wellness, Foundation and the Packard Foundation, January 1996, San Diego.

Sharon B. Buchbinder and Anne K. Duggan "Evaluation of the Hawaii Healthy Start Program," at the Ninth Annual Research Conference, The Research and Training Center for Children's Mental Health, February 1996, Tampa, FL.

Anne K. Duggan and Sharon B. Buchbinder, "Hawaii's Healthy Start Home Visiting Program: Engagement of At-Risk Families," at the Ambulatory Pediatric Association Annual Meeting, May 1996, Washington.

Anne K. Duggan, et al., "Comprehensive Evaluation of Hawaii's Healthy Start Program: Process and Outcomes of Home Visiting for At-Risk Families of Newborns," at the 11th International Conference on Child Abuse and Neglect, ISPCAN (International Society for Prevention of Child Abuse and Neglect), August 1996, Dublin, Ireland.

Anne K. Duggan, et al., "Evaluation of the Hawaii Healthy Start Program," at the 11th National Conference on Child Abuse and Neglect, NCCAN (National Center on Child Abuse and Neglect), September 1996, Washington.

Anne K. Duggan, et al., "Comprehensive Evaluation of Hawaii's Healthy Start Program: Family Engagement at One Year," at the Annual Meeting of the American Public Health Association, November 1996, New York.

Anne K. Duggan, et al., "Randomized Trial of Hawaii's Healthy Start Program: Family Engagement and Outcomes at Year One," at the Western Regional Within Our Reach Conference, Effective Home-Based Strategies for Family Support, sponsored by the California Department of Social Services, the California Department of Health, the Stuart Foundation, the California Wellness Foundation and the Packard Foundation, April 1997, Sacramento, CA.

Anne K. Duggan, et al., "Randomized Controlled Trial of Hawaii's Home Visiting Program: Parenting Outcomes at One Year," at the joint APS (American Pediatric Society)/SPR (Society of Pediatric Research)/APA (American Pediatric Association) Platform Session, annual meeting of the Pediatric Academic Societies, May 1997, Washington.

Anne K. Duggan, et al., "Randomized Controlled Trial of Hawaii's Home Visiting Program for At-Risk Families: Parenting Outcomes at One Year," at the Annual Meeting of the American Public Health Association, November 1997, Indianapolis.

Anne K. Duggan, et al., "Randomized Controlled Trial of Hawaii's Home Visiting Program for At-Risk Families: Childhood Injury Prevention," at the Annual Meeting of the American Public Health Association, November 1997, Indianapolis.

Amy M. Windham et al., "Randomized Controlled Trial of Hawaii's Home Visiting Program: Effects on Maternal Mental Health," at the Annual Meeting of the American Public Health Association, November 1997, Indianapolis.

Anne K. Duggan, et al., "Developing Optimal Models for Family Support: What Research Is Telling Us," at the 12th Annual San Diego Conference on Responding to Child Maltreatment, Center for Child Protection, January 1998, San Diego.

Anne K. Duggan, et al., "Involvement of the Entire System in the Home Visiting System," at the Western Regional Within Our Reach Conference, Effective Home-Based Strategies for Family Support, sponsored by the California Department of Social Services, the California Department of Health, the Stuart Foundation, the California Wellness Foundation and the Packard Foundation, April 1998, San Francisco.

Anne K. Duggan, et al., "Setting Up Quality Improvement Programs, Blueprints for Change," at the Western Regional Within Our Reach Conference, Effective Home-Based Strategies for Family Support, sponsored by the California Department of Social Services, the California Department of Health, the Stuart Foundation, the California Wellness Foundation and the Packard Foundation, April 1998, San Francisco.

Anne K. Duggan, et al., "Home Visitation: Does It Change Children's Lives?," at the American Academy of Pediatrics Spring session, April 1998, Atlanta.

Anne K. Duggan, et al., "Randomized Controlled Trial of Hawaii's Home Visiting Program for At-Risk Families: Impact on Fathers' Involvement at One Year," at the Annual Meeting of the American Pediatric Societies, May 1998, New Orleans.

Amy W. Windham, et al., "Comprehensive Evaluation of the Hawaii Healthy Start Program: Effects on Child Abuse and Neglect During the First Year of Life," at the Annual Meeting of the American Pediatric Societies, May 1998, New Orleans.

Amy M. Windham, et al., "Comprehensive Evaluation of the Hawaii Healthy Start Program: Effects on Maternal Mental Health and Substance Abuse at One Year," at the Annual Meeting of the National Committee on the Prevention of Child Abuse, May 1998, Chicago.

Anne K. Duggan, et al., "Comprehensive Evaluation of the Hawaii Healthy Start Program: Effects on Maternal Parenting Behavior in the First Two Years of Life," at the 12th International Conference on Child Abuse and Neglect, ISPCAN (International Society for the Prevention of Child Abuse and Neglect), September 1998, Auckland, New Zealand.

Anne K. Duggan, et al., "Going to Scale: Quality Assurance in a Statewide Prevention Program," at the 12th National Conference on Child Abuse and Neglect, November 1998, Cincinnati.

Anne K. Duggan, et al., "Randomized Controlled Trial of Hawaii's Home Visiting Program for At-Risk Families: Impact of the Father's Involvement in Parenting," at the Annual Meeting of the American Public Health Association, November 1998, Washington.

Anne K. Duggan, "What Can We Say About Home Visiting?, at the Workshop on Home Visitation Interventions" sponsored by the National Academy of Sciences' Institute of Medicine and National Research Council, March 8–9, 1999, Washington.

Elizabeth McFarlane, et al., Randomized Controlled Trial of Hawaii's Home Visiting Program for At-Risk Families: Maternal Parenting Behavior and Child Development Outcomes at Two Years, at the Annual Meeting of the American Public Health Association, November 1998, Washington.

Elizabeth McFarlane, et al., "Addressing Child Development Needs in Paraprofessional Home Visitation Programs: Results and Recommendations from the Comprehensive Evaluation of the Hawaii Healthy Start Program," at the Western Regional Within Our Reach Conference, Effective Home-Based Strategies for Family Support, sponsored by the California Department of Social Services, the California Department of Health, the Stuart Foundation, the California Wellness Foundation and the Packard Foundation, April 1999, Los Angeles.

Amy M. Windham, et al., "Addressing Mental Health Problems in a Paraprofessional Home Visitation Program: Results and Recommendations from the Comprehensive Evaluation of the Hawaii Healthy Start Program," at the Western Regional Within Our Reach Conference, Effective Home-Based Strategies for Family Support, sponsored by the California Department of Social Services, the California Department of Health, the Stuart Foundation, the California Wellness Foundation and the Packard Foundation, April 1999, Los Angeles.

Amy M. Windham, et al., "Malleable Risk Factors for Child Maltreatment: Opportunity for Pediatric Intervention," at the Annual Meeting of the American Pediatric Societies, May 1999, San Francisco.

Elizabeth McFarlane, et al., "Does Home Visiting Improve Pediatrician's Recognition of Maternal Risks and Child Developmental Delay? Results and Recommendations of Hawaii's Health Start Program," at the Western Regional Within Our Reach Conference: Effective Home-Based Strategies for Family Support, April 2000, Los Angeles.

Loretta Fuddy, et al., "Evaluating Hawaii's Health Start: Lessons Learned and Next Steps," at the Western Regional Within Our Reach Conference: Effective Home-Based Strategies for Family Support, April 2000, Los Angeles.

Anne K. Duggan, "Evaluation of Hawaii's Health Program," at the Twenty-ninth Research Roundtable, Maternal and Child Health Bureau, July 2000, Washington.

Anne K. Duggan, "Evaluation of Medical Homes," at the Maternal and Child Health Bureau/American Academy of Pediatrics National Conference on the Medical Home, January 2001, Honolulu.

 Back to the Table of Contents


Report prepared by: Antonia Sunderland
Reviewed by: Robert Crum
Reviewed by: James Wood
Program Officer: Seth Emont
Program Officer: Michael Beachler

Most Requested