December 2007

Grant Results

National Program

Changes in Health Care Financing and Organization

SUMMARY

Researchers at the University of Southern Maine examined the feasibility of using physician profiling software systems to rank physician specialists by their "cost efficiency" — that is, the extent to which the actual costs incurred by their patients are greater or less than expected for those patients, on average.

The researchers assessed whether two commercially available software packages could produce consistent rankings for each specialty:

  • When the two packages were compared one to the other.
  • When the rankings were compared year to year.

This project was part of the Robert Wood Johnson Foundation (RWJF) national program Changes in Health Care Financing and Organization (HCFO) (for more information see Grant Results).

A Findings Brief on the project is available on the HCFO Web site.

Key Findings

  • Agreement between rankings from consecutive years was substantial for cardiology, general surgery and neurology, but less so for primary care specialties such as family practice, internal medicine and gynecology.
  • Agreement between rankings using the two software systems was also substantial for cardiology, general surgery and neurology. There was little agreement between rankings for family practice, foot surgery, internal medicine and gynecology.
  • If pharmacy claims data are absent, accurate rankings are still feasible for cardiology, general surgery and neurology, but not for family practice.

Key Conclusions

  • With the use of appropriate methodologies, health plans and patients can feel confident of cost-efficiency rankings for specialists such as cardiologists, general surgeons and neurologists.
  • Caution is needed when interpreting rankings for others, such as family practitioners, gynecologists, internists and foot surgeons.
  • Although cost-efficiency rankings might still be useful for feedback to physicians and for improving practice management, using inaccurate scores to reward or penalize physicians should be avoided.

Funding
RWJF supported the project from June 2003 to June 2004 with a grant of $102,828 to the University of Southern Maine.

 See Grant Detail & Contact Information
 Back to the Table of Contents


THE PROJECT

Managed care health plans, physician organizations, Medicaid programs and other groups use commercially available physician profiling systems to assess a physician's cost efficiency.

Cost efficiency is the extent to which the health care costs incurred by a physician's patients are more or less than the costs expected for those patients on average, given their diagnoses, health statuses and demographic characteristics (age, gender, etc.).

Health plans use these assessments to rank physicians for cost efficiency as they make decisions about which physicians to include in their networks. In addition, some health plans offer financial incentives to patients who select "efficient" providers.

Purchasers can compare systems on price, types of reports produced and hardware needed, but whether these profiles are reliable indicators of cost efficiency remains unclear.

Under an earlier grant (ID# 036874) from RWJF's Health Care Financing and Organization (HCFO) national program, researchers at the University of Michigan School of Public Health evaluated seven methodologies for profiling the cost efficiency of primary care physicians. They found that primary care physician rankings developed using the different methodologies were very consistent.

Given this result, researchers sought to test the feasibility of using profiling systems to evaluate the cost efficiency of physician specialists.

The researchers assessed whether two commercially available software packages could produce consistent rankings for each specialty:

  • When the two packages were compared one to the other.
  • When the rankings were compared year to year.

RWJF originally made this grant to the University of Michigan School of Public Health (as grant ID# 047781) in March 2003, under the co-direction of Kyle Lynn Grazier, Ph.D., and J. William Thomas, Ph.D. Before the project began, Thomas moved to the University of Southern Maine (Edmund S. Muskie School of Public Service), and the project moved with him as principal investigator. Grazier continued to be involved with the project as a consultant.

Methodology

Researchers analyzed four years (1999 through 2002) of claims data (professional, inpatient, outpatient and pharmacy) for nine specialties from a university-owned health maintenance organization (HMO) in Michigan. The "mixed-model" HMO featured primary care and specialist physicians who practiced in a group setting or as part of an independent practice association.

The nine specialties were:

  • Cardiology
  • Family practice
  • Foot surgery
  • General internal medicine
  • General surgery
  • Gynecology
  • Neurology
  • Orthopedic surgery
  • Urology

The investigators used two commercially available software systems to organize claims data so that multiple visits for short-duration conditions (such as acute bronchitis) and chronic conditions (such as congestive heart failure or emphysema) could be grouped into "episodes" of care. The software grouped the data into episodes by diagnosis and by time period.

The analysis of the episode data included several parts:

  • Since pharmacy claims are not always available, researchers examined whether lack of pharmacy claims data would affect physician cost-efficiency rankings. To do this, they analyzed the same episode data, both with and without pharmacy claims.
  • Researchers processed the episode data through commercially available risk-adjustment software that calculates a health risk score that is predictive of health care cost. It also provides a "retrospective risk score," which explains variation in current costs, and a "prospective risk score," which predicts future costs. For cost-efficiency profiling, the researchers (and others) use the retrospective risk score.
  • Researchers tested three statistical methods for handling patients/episodes that have especially high costs ("outliers").
  • Researchers also tested three "attribution rules" in order to assign financial responsibility for the cost of each episode of care to a particular physician, depending on his or her share of the total cost of care for each episode.

 Back to the Table of Contents


FINDINGS

The researchers reported findings in articles in the American Journal of Managed Care (article available online), Health Services Research (abstract available online) and Inquiry (abstract available online), in a Findings Brief available online at the HCFO Web site and in a report to RWJF. (See the Bibliography.)

  • Agreement between rankings from consecutive years was substantial for cardiology, general surgery and neurology. There was less consistency in rankings for primary care specialties, including family practice, internal medicine and gynecology. (Findings Brief)
  • Agreement of the two episode-grouping software systems on physician practice efficiency rankings differed by specialty (report to RWJF):
    • Rankings of cardiologists, general surgeons and neurologists were in substantial agreement in most cases.
    • There was little agreement between rankings for family practitioners, foot surgeons, general internists and gynecologists.
  • If pharmacy claims data are absent, accurate rankings are still feasible for cardiology, general surgery and neurology, but not for family practice. Differences by specialty depend largely on the proportion that pharmacy costs make up in the total cost of an episode of care. On average, pharmacy costs for family practice are a significantly higher portion of total episode costs, compared with the other specialties. (American Journal of Managed Care and Findings Brief)
  • Additional adjustments for patient health risks do not appear to increase the accuracy of physician profiles in four specialties studied (cardiology, family practice, general surgery and neurology). Risk scores were mostly unrelated to episode costs in about 75 percent of episode categories and showed limited relation in most of the others. (Health Services Research)
  • None of the three outlier methods nor the three attribution rules tested was associated with consistently more reliable rankings than any of the others. (Inquiry and report to RWJF)
  • The minimum number of episodes of care that qualifies a physician for profiling can have a strong effect on the accuracy of the resulting profiles. The percentages of physicians eligible for profiling varied greatly by specialty. For four specialties, fewer than 10 percent of physicians were eligible when the minimum number of episodes per physician was set at 25. (Findings Brief and report to RWJF)

 Back to the Table of Contents


CONCLUSIONS

  • With use of appropriate methodologies, health plans and patients can feel confident of cost-efficiency rankings for specialists such as cardiologists, general surgeons and neurologists.
  • Caution is needed when interpreting rankings for others, such as family practitioners, gynecologists, internists and foot surgeons.
  • Although cost-efficiency rankings might still be useful when used for feedback to physicians and for improving practice management, using inaccurate scores to reward or penalize physicians should be avoided.

 Back to the Table of Contents


GRANT DETAILS & CONTACT INFORMATION

Project

Using Physician Profiling Software to Evaluate the Practice Efficiency of Physician Specialists

Grantee

University of Southern Maine (Portland,  ME)

  • Using Primary Care Physician Profiling Software to Access the Performance of Physician Specialists
    Amount: $ 102,828
    Dates: July 2003 to June 2004
    ID#:  048779

Contact

J. William Thomas, Ph.D.
(207) 529-2009
jwthomas@usm.maine.edu

Web Site

http://www.hcfo.net/about.htm

 Back to the Table of Contents


BIBLIOGRAPHY

(Current as of date of this report; as provided by grantee organization; not verified by RWJF; items not available from RWJF.)

Articles

Thomas JW. "Economic Profiling of Physicians: Does Omission of Pharmacy Claims Bias Performance Measurement?" American Journal of Managed Care, 12(6): 341–351, 2006. Available online.

Thomas JW. "Should Episode-Based Economic Profiles Be Risk Adjusted to Account for Differences in Patients' Health Risks?" Health Services Research, 41(2): 581–598, 2006. Abstract available online. Full text requires subscription or fee.

Thomas JW and Ward K. "Economic Profiling of Physician Specialists: Use of Outlier Treatment and Episode Attribution Rules." Inquiry, 43(3): 271–282, 2006. Abstract available online.

 Back to the Table of Contents


Report prepared by: Mary B. Geisz
Reviewed by: Richard Camer
Reviewed by: Molly McKaughan
Program Officer: Nancy Barrand

Most Requested