January 2005

Grant Results

SUMMARY

During 2000 and 2001, researchers at the Foundation for Accountability (FACCT, no longer in existence) conducted a series of surveys to evaluate the feasibility of using Internet-based surveys of consumers to develop quality ratings of physicians, hospitals, health plans and other health care providers.

The surveys:

  • Compared responses from consumers who rated individual physicians and practices in a mailed survey with responses from those who rated the same physicians and practices via the Internet.
  • Assessed various methods of recruiting consumers to participate in such surveys.
  • Measured the level of satisfaction among users of FACCT's Compare Your Care Internet survey tool.
  • Probed nonusers about their lack of participation.

Key Findings
In a report to the Robert Wood Johnson Foundation (RWJF), FACCT's findings included:

  • There was little difference in the average ratings of individual physicians or practices provided by consumers who responded by mail and those who responded via the Internet.
  • Response rates varied widely but were highest among potential respondents recruited through so-called trusted sources, such as physicians and consumer groups.
  • Most respondents said they would use comparative ratings and comments from other patients to choose a doctor.
  • Nonusers cited lack of Internet access or time as their primary reason for not using the tool.

Funding
RWJF supported this project through a grant of $394,584.

 See Grant Detail & Contact Information
 Back to the Table of Contents


THE PROJECT

This grant from RWJF to FACCT supported a project that evaluated the feasibility of using Internet-based surveys to develop quality ratings of physicians, hospitals, health plans, and other health care providers. FACCT is a nonprofit organization that acts to develop tools, measures, and initiatives that help better inform consumers in their health care decisions. In previous support from RWJF, FACCT developed measures concerning health system performance in pediatric care, end-of-life care, and early identification and reduction of alcohol-related problems. (See Grant Results ID# 031845 on pediatric end-of-life care and ID# 030715 on reduction of alcohol-related problems.)

To organize local involvement, identify community partners, and establish an advisory committee, FACCT contracted with the Maine Health Management Coalition, a coalition of 30 employers and the Maine Medicaid program, which together provide health insurance coverage for more than 300,000 people. The advisory committee comprised some two dozen individuals, representing physicians, health plans, consumer groups, and employers. A smaller work group reviewed survey questions, scoring algorithms, and the feedback offered to consumers during the project.

Under the grant, FACCT conducted a series of patient surveys to test the feasibility of collecting information from consumers via the Internet.

  • The first survey was designed to test whether information obtained from consumers differs markedly if obtained on-line rather than through a mail survey. Respondents were recruited from the patient rosters of nine physicians in two Maine practices. Surveys were sent to 1,500 potential on-line respondents and 800 potential mail respondents. FACCT contracted with researchers at the Health Institute of the New England Medical Center in Boston, Mass., to analyze the survey responses.
  • In the second survey, FACCT explored the effectiveness of using different sources and methods to recruit potential users of on-line health care information tools. The survey was based on FACCT's Compare Your Care (CYC) Internet tool, which helps users assess the quality of health care they receive. Survey participants were recruited from two Maine medical practices, a New York union, a non-profit pension and investment management organization, and a national patient advocacy organization. (See Appendix 1 for details.) The survey was originally planned to test the feasibility of developing provider-specific quality ratings based on on-line survey responses by consumers in Augusta, Maine, but those plans were abandoned after physician members of the advisory group expressed concerns that such ratings could be publicly reported.
  • In a follow-up survey, FACCT contacted CYC users by e-mail one to two weeks after the users used the tool. About half (54.4 percent) of the 515 people who used the site had left their e-mail address so they'd be contacted later; about a third (38 percent) of those people responded to the follow-up survey.
  • In a second follow-up survey, FACCT contacted New York union members who'd declined to use the on-line survey tool. The survey was administered by a union-run polling center. More than 9,800 phone calls were made, 552 individuals were reached, and 100 surveys were completed.

 Back to the Table of Contents


FINDINGS

Survey comparing on-line and mail responses

  • There was no evidence that consumers' ratings of physicians or practices differed significantly whether the groups responded to the survey on-line or via conventional mail. Average ratings for the two medical practices and for each of the nine physicians were the same on the part of consumers who responded on-line and for those who responded by mail.
  • Response rates were higher among those invited to participate by using regular mail (58 percent) compared with those invited to participate on-line (25 percent). However, after adjustments were made for the fact that only half of Maine households have Internet access, the response rate among those in the Internet group approached that of the regular mail group, the investigators say.

Survey comparing recruitment sources and methods

  • E-mail solicitations and communications from so-called trusted sources — particularly consumer groups and physicians resulted in the highest response rates. The percentage of those who responded to the solicitation varied widely by project partner from a low of 1 percent among union members to 23 percent among people invited by the patient advocacy organization. National surveys suggest that the level-of-trust factor is very important, but other factors, such as social and economic background and language skills, may also influence the effectiveness of recruitment efforts.

Follow-up survey of Compare Your Care Users

  • Most (87 percent) respondents said they would use comparative ratings to choose a doctor, and three-quarters (75 percent) said they'd use comments from other patients.

Follow-up survey of nonusers

  • Logistical difficulties were the biggest cause of nonuse. Most nonusers reported lack of access to a computer or the Internet (63 percent) or lack of time (30 percent). Only 2 percent were concerned about privacy, and 5 percent cited lack of interest in the tools.

Communications

FACCT completed a 40-page report on the project and distributed it to 400 consumer organizations, unions, purchasers, and researchers. The report was also placed on the organization's Web site. Brief articles describing the project appeared in Health Affairs and Managed Care Interface. Investigators are continuing work on an article to be submitted for publication in a professional journal. (See the Bibliography.)

 Back to the Table of Contents


LESSONS LEARNED

  1. Any project that seeks to develop provider-specific ratings must first identify a supportive community ready to champion the effort. Physician members of the advisory group expressed strong reservations about the development of provider-specific ratings, forcing the project to abandon plans to undertake a community-level project in Augusta, Maine. Any future efforts to undertake such a project will require identification of a local champion, engagement of the provider community as a key partner, and extensive communication during the early stages of the project.

 Back to the Table of Contents


AFTER THE GRANT

In January 2002, FACCT began field work on a project with investigators at Brigham and Women's Hospital in Boston, Mass., that will compare self-ratings obtained with FACCT's online survey tool with data obtained from patients' actual medical charts.

 Back to the Table of Contents


GRANT DETAILS & CONTACT INFORMATION

Project

Examining the Validity and Usefulness of the Internet as a Tool for Collecting and Providing Health Information

Grantee

Foundation for Accountability (FACCT)

  • Amount: $ 394,584
    Dates: June 2000 to May 2001
    ID#:  038238

Contact

Elizabeth Whitworth
(503) 223-2228
ewhitworth@facct.org

Web Site

http://www.facct.org

 Back to the Table of Contents


APPENDICES


Appendix 1

(Current as of the time of the grant; provided by the grantee organization; not verified by RWJF.)

Recruitment Study Partners

American Diabetes Association
Alexandria, Va.

1199 National Benefit Fund
New York, N.Y.

Maine General Health Associates
Gardiner, Maine

Martin's Point Health Care
Portland, Maine

Teachers Insurance and Annuity Association - College Retirement Equities Fund
New York, N.Y.

 Back to the Table of Contents


BIBLIOGRAPHY

(Current as of date of this report; as provided by grantee organization; not verified by RWJF; items not available from RWJF.)

Reports

Report and Key Findings from the FACCT's Quality Counts Initiative, June 2000 to May 2001. Portland, Ore.: FACCT, 2001.

 Back to the Table of Contents


Report prepared by: Sean Gilbert
Reviewed by: Richard Camer
Reviewed by: Molly McKaughan
Program Officer: Robin E. Mockenhaupt