September 2006

Grant Results

SUMMARY

During 2000 and 2001, staff at the New Jersey Department of Health and Senior Services surveyed patients who belong to New Jersey health maintenance organizations (HMOs) and use behavioral health services, to assess the use and quality of those services.

Key Findings

  • Respondents gave their highest ratings to quality of care or counseling services and their lower ratings to the health plan overall.
  • Persons who received out-of-network services were more positive about their care than were patients who received in-network services, but less satisfied with the insurer's administrative process.
  • Almost half of all users of behavioral health services indicated that they are "much better" than they were before treatment.

Funding
The Robert Wood Johnson Foundation (RWJF) supported this project through a grant of $225,395.

 See Grant Detail & Contact Information
 Back to the Table of Contents


THE PROJECT

Health care consumers use behavioral health services offered by health plans, but information on the consumer's perspective on service quality has been lacking. To assess the use and quality of those services, the New Jersey Department of Health and Senior Services surveyed patients who were commercially insured by New Jersey health maintenance organizations (HMOs) and using behavioral health services.

RWJF provided grant support for the effort. The project was the first statewide survey to collect behavioral health information on a plan-specific basis. It was also one of the first wide-scale uses of ECHO, a survey in the public domain, and thus provided an opportunity for its developers, a research team from Harvard University, to further refine the instrument. (In exchange for data from the project, the Harvard team agreed to provide technical support.)

The Harvard team based ECHO on CAHPS, another survey it had developed, which is the recognized standard for assessing consumer experiences with medical care. Since 1997, the New Jersey Department of Health and Senior Services has used CAHPS to report publicly on consumer experiences with medical care through its annual New Jersey HMO Performance Report. The developers designed this project to extend public reporting to include behavioral health services.

The grantee organization contracted with an independent survey organization, Eagleton Institute of Politics, Center for Public Interest Polling, at Rutgers University, New Brunswick, N.J., to collect the data. An advisory workgroup helped address technical and policy issues, with members representing HMOs, managed behavioral health organizations (MBHOs), behavioral health providers and other interested parties.

The RWJF grant supported advisory group meetings, survey implementation and data analysis. The New Jersey Department of Health and Senior Services provided management, policy and administrative support ($100,000+). In the interest of further developing ECHO, the Harvard Medical School team provided the survey instrument and extensive technical assistance to design the research and analyze the data.

The New Jersey Department of Health and Senior Services required participation of the state's seven largest HMOs, accounting for over 95 percent of all enrollees in commercial managed care in the state (Aetna-US Healthcare, AmeriHealth, Cigna, Horizon, Oxford, Physicians Health Services [now Health Net] and United Healthcare).

Respondents included all recipients of mental health, drug and alcohol counseling for both in- and out-of-network providers, combined with a smaller sample of patients not using those services (non-users). Researchers selected 8,993 respondents for the study — 7,131 recipients of behavioral health services in the previous 12 months and 1,862 non-users. Data collection took place from April through June 2001.

As detailed in the findings below, researchers compared data from ECHO to the results of the 2001 CAHPS, which had been fielded with a separate sample. The ECHO survey measures respondents' experiences with behavioral health care, including both overall evaluations of care and specific experiences, in the following areas:

  1. Patterns of access and utilization.
  2. Satisfaction with counseling and treatment.
  3. Member satisfaction with involvement in individual treatment plans.
  4. Experiences with MCO administrative actions (e.g., claims processing).
  5. Perceived improvement in mental health status.

 Back to the Table of Contents


FINDINGS

The New Jersey Department of Health and Senior Services included the following key findings in a survey report, Review of Behavioral Health Services Provided by New Jersey HMOs: Implementation of the ECHO Survey:

  • Comparison of respondent characteristics: ECHO respondents were younger, more likely to be female, highly educated, white and to self-report poorer physical health than those measured by the 2001 CAHPS.
  • Statewide plan averages: ECHO respondents gave their highest ratings to quality of care or counseling services and their lower ratings to the health plan overall. For example, 32 percent rated quality of care highly (a 9 or 10 on a scale from 0 to 10), 45 percent rated counseling highly and 24 percent rated the health plan highly.
  • Satisfaction with health plan: CAHPS respondents were more satisfied with their plans in regard to physical health than ECHO respondents were with behavioral health. For example, 24 percent of plan members rated their plan highly (a 9 or 10 on a scale from 0 to 10) in the ECHO survey compared to 32 percent of members in the CAHPS survey.
  • Satisfaction with care: Plan members rated their behavioral health care and their physical health care similarly, as measured by the ECHO and the CAHPS.
  • Out-of-network versus in-network services: Persons who received out-of-network services were more positive about their care than were patients who received in-network services, but were less satisfied with the insurer's administrative process. For example, 55 percent of out-of-network respondents rated their counseling highly (a 9 or 10 on a scale from 0 to 10) as compared with 42 percent of in-network users.
  • Addictions treatment versus behavioral health treatment: Persons who received treatment for drug or alcohol addiction (6 percent of total) were more likely than persons who received other forms of behavioral health treatment to be dissatisfied with their counselors and had a very different view of the treatment process. According to the principal investigator, this is a small subgroup, so caution in interpreting this data is in order. She also noted that the finding warrants further investigation.
  • Perceived improvement: Almost half of all users of behavioral health services indicated that they are "much better" than they were before treatment.

Due to significant variation in the sample selection process used by the different health plans, the project did not report results for individual plans, as originally envisioned.

Limitations

  • Because the ECHO survey is based on different groups who have different experiences in obtaining behavioral health services — including people using in- and out-of-network providers and both mental health and addictions therapy — subsequent analysis will be needed to distinguish further among their views. For example, enrollees with a point-of-service plan have the option of choosing a provider outside the network. The MCO has no responsibility for the quality of these out-of-network services. In contrast, the MCO is responsible for quality oversight among its in-network services.
  • A comparison of CAHPS and ECHO study results indicates these two groups have different demographic characteristics (age, gender, education, etc.). Any comparison of the results of the two surveys must take these limitations into consideration.

Communications

The Eagleton Institute outlined the survey results in a technical report, 2001 New Jersey Experience of Care and Health Outcomes (ECHO™) Survey. Using the survey data, Harvard Medical School researchers produced a technical paper and contributed a chapter to a book. (See Bibliography for details.) They also presented the findings to the nonprofit, Washington, D.C.-based National Committee on Quality Assurance (NCQA), the accrediting organization for MCOs.

The New Jersey Department of Health and Senior Services published an eight-page report of survey findings, Review of Behavioral Health Services Provided by New Jersey HMOs: Implementation of the ECHO Survey, disseminating it along with the 2002 New Jersey HMO Performance Report. The document can be found online.

 Back to the Table of Contents


LESSONS LEARNED

  1. More time should be built into project timelines in order to resolve issues related to data provided by health plans. Varying data procedures among the MCOs created problems for the researchers in determining accurate enrollment levels and correct identification of those clients who received behavioral health treatment. (Principal Investigator)

 Back to the Table of Contents


AFTER THE GRANT

Harvard Medical School investigators are continuing to use the data to refine ECHO further. In 2003, NCQA implemented the national ECHO survey as a HEDIS® measure that is required for MBHO accreditation, the protocols for which are based on those developed for this project. The protocols are published in volume 7 of HEDIS®; 2003, "Specifications for the ECHOTM 1.1H Survey for MBHOs".

 Back to the Table of Contents


GRANT DETAILS & CONTACT INFORMATION

Project

Assessing Patient Perceptions of Behavioral Health Services in New Jersey HMOs

Grantee

State of New Jersey Department of Health and Senior Services (Trenton,  NJ)

  • Amount: $ 225,395
    Dates: July 2000 to December 2001
    ID#:  037698

Contact

Frances Prestianni, Ph.D.
(609) 984-7914
frances.prestianni@doh.state.nj.us

 Back to the Table of Contents


BIBLIOGRAPHY

(Current as of date of this report; as provided by grantee organization; not verified by RWJF; items not available from RWJF.)

Book Chapters

Daniels AS, Shaul JA, Greenberg P and Cleary PD. "The Experience of Care and Health Outcomes Survey (ECHO™): A Consumer Survey to Collect Ratings of Behavioral Health Care Treatment, Outcomes, and Plans." In The Use of Psychological Testing for Treatment Planning and Outcomes Assessment, Third Edition. Maruish ME (ed.). Mahwah, N.J.: Lawrence Erlbaum Associates, 2004–05.

Articles

Shaul JA, Cubanski J, Eisen SV and Cleary PD. "Psychometric Properties of the Experience of Care and Health Outcomes (ECHO) Survey Measures." Unpublished.

Reports

Eagleton Institute of Politics, Center for Public Interest Polling. 2001 New Jersey Experience of Care and Health Outcomes (ECHO) Study. New Brunswick, N.J.: Eagleton Institute, 2001.

New Jersey Department of Health and Senior Services. Review of Behavioral Health Services Provided by New Jersey HMOs: Implementation of the ECHO Survey. Trenton, N.J.: New Jersey Department of Health and Senior Services, 2002. Also appears online.

Survey Instruments

Eagleton Institute of Politics, Center for Public Interest Polling and State of New Jersey, Department of Health and Senior Services, Experience of Care and Health Outcomes Survey. Fielded statewide in New Jersey. April–June 2001.

 Back to the Table of Contents


Report prepared by: Jan Hempel
Reviewed by: Robert Crum
Reviewed by: Marian Bass
Program Officer: Constance M. Pechura

Most Requested