August 2002

Grant Results

SUMMARY

The Parents as Teachers National Center (PATNC) conducted a multi-site randomized evaluation starting in 1997 of outcomes for low-income infants, toddlers and parents who received services from Parents as Teachers (PAT).

PAT, the nation's largest home-visiting program, provides parents with support and information on their children's development.

PATNC coordinated the evaluation and sub-contracted with SRI International, an independent, nonprofit, knowledge-based research and consulting organization in Menlo Park, Calif., for design, data collection and analyses.

During the third and fourth years of the project, PATNC organized a "Forum on Home Visiting" in Washington, D.C. and began a qualitative study of recruitment retention and program improvement.

Key Results

  • SRI completed the three-site randomized evaluation of PAT's outcomes through the child's third birthday at one site and second birthday at the other two sites.
  • PATNC sponsored the "Forum on Home Visiting" on Dec. 13–14, 1999, in Washington, D.C.
  • SRI and PATNC conducted a qualitative study of a subset of high needs families that participated in PAT for at least one year.

Key Findings
According to reports submitted to the Robert Wood Johnson Foundation (RWJF) by PATNC and SRI:

  • PAT is most effective when it is part of an umbrella of social services.
  • PAT had some positive benefits to parents in improving their knowledge, attitudes and behaviors. Families with the lowest incomes ($15,000 or less annually) benefited more from PAT than other families.
  • PAT had a small positive effect on the children's social adjustment.
  • PAT had a few health benefits.
  • People should have modest expectations for PAT and other home visiting programs. There are many levels of parental engagement in home visiting programs. There is no single profile of an engaged parent. Some assumptions regarding barriers to engagement appear to be inaccurate.
  • Some parent educator qualities are critical to engaging parents successfully.

Limitations

  • Because the sample size was so much smaller than planned and differences between the sites were great, the overall quantitative evaluation results should be viewed with caution.

Funding
RWJF provided partial funding for this project with a grant of $431,492 from February 1997 to July 2001.

 See Grant Detail & Contact Information
 Back to the Table of Contents


THE PROBLEM

A number of studies show that interventions to ensure that children are raised in nurturing families prevent harmful behaviors such as substance abuse, teen pregnancy and exposure to HIV infection (Olds, 1988; Seitz, 1985; Johnson, 1987; Lally, 1988; and Schweinhart, 1993), and that public investments to strengthen families' child-rearing capacities reduce social costs (Barnett, 1993).

PAT, the nation's largest home-visiting program, provides parents with support and information on their children's development. Initiated by the state of Missouri's Department of Elementary and Secondary education, the PAT program has grown from four pilot sites in 1981 to more than 2,800 programs in 50 states and five foreign countries in 2002.

The program, which serves volunteer families with relatively high levels of motivation to participate, has four components:

  1. home visits by trained and certified family educators at least monthly
  2. periodic group meetings of parents
  3. regular screenings of the children's health and development
  4. links to medical and social support services.

Previous evaluations of PAT were generally small, unscientific studies. Recognizing the need for definitive research on the program's impact, PATNC convened three expert panels in 1994 and 1995. These panels recommended that PATNC conduct a full-scale randomized evaluation of PAT's effects and a follow-up qualitative study including low-income minority families. PATNC planned a three-phase study:

  1. design
  2. implementation
  3. follow-up.

Phase 1 ran from July 1995 to June 1996 and covered site selection, additional literature review, final design specification, choosing study instruments, developing a practitioner's guidebook on evaluation and consulting with an advisory group. PATNC sub-contracted these activities to SRI International, an independent, nonprofit, knowledge-based research and consulting organization in Menlo Park, Calif. The Carnegie Corporation and the Ewing Marion Kauffman Foundation funded Phase 1.

 Back to the Table of Contents


THE PROJECT

By January 1997, PAT was underway in 1,920 sites and policymakers were looking at the program as a potential model for helping distressed families nurture their children. This grant from RWJF provided partial funding to PATNC to conduct a multi-site randomized evaluation of PAT's outcomes for low-income infants, toddlers and parents (Phase 2). The original objectives were to assess the degree to which PAT improves: (1) parental knowledge, behaviors and skills in creating a safe and nurturing home for their children; and (2) children's health status and their cognitive and social development, including parent-child interactions.

PATNC coordinated the evaluation and sub-contracted with SRI International for data collection and analyses. The Office of Educational Research and Improvement of the US Department of Education was the lead funder for this project ($664,209); the Smith Richardson Foundation also supported the project ($268,115).

SRI planned to evaluate 1,250 families at three PAT programs. To retain confidentiality, the sites are not named publicly. A fourth site was dropped from the study shortly after the recruitment phase when the local PAT program was closed for six months due to the financial problems of its host school district. During the first two years of the grant, SRI recruited 667 families into the study. Each family had either a pregnant woman or a child under the age of 8 months. Some 80 percent of these families reported incomes of less than $25,000; almost 60 percent reported annual incomes of less than $15,000.

They were divided randomly into two groups: (1) a participant group (272 families) that would receive monthly home visits and other PAT services; and (2) a control group (395 families) that would receive only the services that were normally available in their community and that they requested. SRI planned to measure parent and child outcomes through the child's third birthday, using 14 data collection instruments (including questionnaires, surveys, rating scales and program records). Field evaluators collected most of these data through in-home assessments at or around the child's birthday.

Because of the large number of low-income families required for the evaluation, however, SRI had to actively recruit families that might not have otherwise sought PAT services. These families were poorer and less motivated to participate than typical PAT families. As a result, nearly 58 percent of the families in the participant group had dropped out of the study by the end of the second year of the project, and it was no longer possible to produce statistically significant results in two of the sites, where the evaluation was stopped after the two-year assessment.

According to the project director, the attrition of recruited participants was not entirely due to a lack of motivation, but could be attributed to the recruitment strategy. Recruited families were randomly assigned into one of two groups: one receiving home visits and the other receiving yearly gift certificates. The project director says that many in the group receiving home visits never wanted to be in the treatment group, but were probably more interested in receiving gift certificates. The evaluation at one site, where the program was part of an umbrella of community resources, continued as planned.

PATNC redirected the third and fourth years of the project to explore the issues of recruitment, retention and engagement (the extent to which parents are actively involved) in PAT, and the quality of program implementation. The new objectives were to: (1) organize a forum of leading home visiting programs, researchers and funders; and (2) begin a qualitative study of the families and parent educators involved in the original study. Some RWJF funds were used to meet the new objectives.

 Back to the Table of Contents


RESULTS

  • SRI completed the three-site randomized evaluation of PAT's outcomes through the child's third birthday at one site, and second birthday in the other two sites.
  • PATNC sponsored the "Forum on Home Visiting" on December 13–14, 1999, in Washington. Some 24 representatives of early childhood home visiting programs, researchers, policymakers and child advocates attended the forum, which addressed recruitment, retention, engagement, working with high-needs families, evaluation and building a research agenda. (High-needs families were defined as those who had a low income, low literacy, history of substance abuse, high mobility, low-birthweight baby and/or receiving public assistance.) Participants agreed to work together to advance the field of home visiting. The Forum, now called the National Forum on Home Visiting has continued to meet since this first meeting.
  • SRI and PATNC conducted a qualitative study of a subset of high-needs families that participated in PAT for at least one year. SRI explored parental engagement in PAT through home-visit observation, interviews with parent educators and PAT site coordinators, and focus groups with parents. PATNC conducted 18 case studies of high-needs families through three interviews with parents and parent educators.
  • SRI wrote four reports detailing the results of: recruitment, the three-year Newark, Del. evaluation, the two-year evaluation for all three sites, and its portion of the qualitative study. See the Bibliography for details.

Findings

According to reports submitted to RWJF by PATNC and SRI:

  • PAT is most effective when it is part of an umbrella of social services. Parents and children at one of the sites, where PAT was part of an umbrella of social services that families used, were more likely to benefit from PAT than parents and children at the other sites.
  • PAT had some positive benefits to parents in improving their knowledge, attitudes and behaviors. Of 25 measures studied, 10 showed small positive effects and 2 (parent's happiness caring for children and behavior that promoted language skills and literacy) showed moderate positive effects.
  • Families with the lowest incomes ($15,000 or less annually) benefited more from PAT than other families. All parents in this group improved their parenting knowledge. Moderate positive effects were noted for 2 of the 14 parent behavior measures.
  • PAT had a small positive effect on the children's social adjustment. At Site 1, children were assessed for three years. Results at age three showed that children in the PAT group were more likely to display positive social and self-help behaviors. The children of PAT teen mothers showed enhanced child development in several areas (academic development, self-help development, social development and physical development)
  • PAT had a few health benefits. PAT children were more likely than others to be fully immunized and less likely to use the emergency room.
  • People should have modest expectations for PAT and other home visiting programs. Dramatic improvements in parents' behavior and children's development from home visiting programs are unlikely, since families with young children face many challenges, and programs such as PAT address complex behavioral and developmental outcomes.
  • There are many levels of parental engagement in home visiting programs. These include: agreeing to and actually participating in home visits, doing the homework between visits and looking for more information on parenting and child development. Parents can be engaged with one measure without being engaged in others.
  • There is no single profile of an engaged parent. For example, better-educated mothers were much more likely than less educated parents to be very involved in their home visits and carry out recommended activities with children between visits, but they were no more likely to keep appointments.
  • Some assumptions regarding barriers to engagement appear to be inaccurate. For example, parent educators reported that parents' employment and school status were each significant barriers to engagement, yet the evaluation found that employment status was not related to whether parents dropped out of PAT, and parents who were in school actually were much more likely to be engaged with PAT.
  • Some parent educator qualities are critical to engaging parents successfully. These included being open to, and accepting of, different family situations and parenting styles, sincerely caring about the parents and children and being able to balance multiple roles. Professional expertise in child development sometimes helped and sometimes hindered engagement.

Limitations

  • Because the sample size was so much smaller than planned and differences between the sites were great, the overall quantitative evaluation results should be viewed with caution.

Communications

PATNC sent the project reports to the funders and also shared the report on engagement with the three sites. PATNC staff made two presentations about the evaluation at the 1997 Parents as Teachers National Center Born to Learn™ International Conference. See the Bibliography for details.

PATNC is writing its qualitative study report, which focuses on four case studies and will be used to train parent educators. PATNC will make the data available to other researchers and will also share the results of the project with its sites and parent educators through newsletters and its Web site.

 Back to the Table of Contents


LESSONS LEARNED

  1. It is difficult to conduct an evaluation with an off-site evaluator. Participating sites had a great deal of extra work and insufficient local support to help them with issues and questions related to participation in the evaluation. (Project Director)
  2. An off-site evaluator needs to collaborate with the sites of the project being evaluated. If sites feel in partnership with the evaluator, challenges of the evaluation can be addressed more effectively. (Project Director)
  3. It is important to consider the community context in which sites operate when choosing sites to evaluate. Parents as Teachers found it difficult to generalize effects across all programs. There were important differences across the three sites. (Project Director)
  4. When recruiting people into a program for the purpose of evaluation, the person making the recruitment call should also make the follow-up call to start participation in the program. Some participants in the Parents as Teachers evaluation resisted the call to schedule a home visit because a different person than the one who recruited them made it. (Project Director)
  5. Projects to help low-income families should consider the significant social isolation that many of these families face. Helping interventions such as home visiting need to take this into account. Parents as Teachers works through the school systems, but low-income families have weak social attachment to schools. (RWJF Program Officer)

 Back to the Table of Contents


AFTER THE GRANT

As a result of this project, PATNC has launched a three-project quality initiative:

  1. developing quality standards and a self-assessment for PAT programs (funded by the David and Lucille Packard Foundation)
  2. building relationships with state affiliates, the organizations that oversee state PAT programs in 28 states (funded by the Ewing Marion Kauffman and Duke foundations)
  3. enhancing the national center's research and evaluation capabilities (funded by the Ewing Marion Kauffman and Duke foundations).

The "Forum on Home Visiting" has been meeting semi-annually. Phase 3 of the original study will not be conducted.

 Back to the Table of Contents


GRANT DETAILS & CONTACT INFORMATION

Project

Multi-Site Evaluation of School-Sponsored Home Visiting Program for Low-Income Families and Infants

Grantee

Parents as Teachers National Center (St. Louis,  MO)

  • Amount: $ 431,492
    Dates: February 1997 to July 2001
    ID#:  028329

Contact

Kate McGilly, Ph.D.
(314) 432-4330
kate.mcgilly@patnc.org

 Back to the Table of Contents


BIBLIOGRAPHY

(Current as of date of this report; as provided by grantee organization; not verified by RWJF; items not available from RWJF.)

Reports

Wagner M. Assessing the Effectiveness of Early Parenting Education and Support Through Home Visiting for Families with Young Children: A Multi-Site Evaluation of the Parents as Teachers Home Visiting Program: Report of Recruitment Results. Menlo Park, Calif.: SRI International, June 1998.

Wagner M, Spiker D, Gerlach-Downie S and Hernandez F. Parental Engagement in Home Visiting Programs — Findings from the Parents as Teachers Multi-site Evaluation. Menlo Park, Calif: SRI International, February 2000.

Wagner M, Spiker D, Hernandez F, Song J and Gerlach-Downie S. Multi-Site Parents as Teachers Evaluation: Experiences and Outcomes for Children and Families. Menlo Park, Calif.: SRI International, June 2001.

Wagner M, Iida E, Spiker D, Hernandez F and Song J. The Multi-Site Evaluation of the Parents as Teachers Home Visiting Program: Three-year Findings from One Community. Menlo Park, Calif.: SRI International, August 2001.

Survey Instruments

SRI International, Parents as Teachers Multi-Site Study: Family Interview, Parents of 1-Year Olds, Ft. Worth, Texas; Newark, Del. and Winston-Salem, N.C. 1998.

Sponsored Conferences

Forum on Home Visiting. December 13–14, 1999,Washington. Attended by 24 representatives of early childhood home-visiting programs, researchers, policymakers and child advocates.

Presentations and Testimony

D Spiker and M Wagner. "Dimensions of Engagement in Home-Visiting Programs: The Parents as Teachers (PAT) Experience," at the Society for Research in Child Development, April 2001, Minneapolis.

 Back to the Table of Contents


Report prepared by: Lori De Milto
Reviewed by: Patricia Patrizi
Reviewed by: Molly McKaughan
Reviewed by: Robert Crum
Program Officer: Terrance Keenan