March 2009

Grant Results

SUMMARY

Researchers from Park Nicollet Institute and the Johns Hopkins Bloomberg School of Public Health conducted preliminary work to develop a standardized set of electronic-health-record-based quality performance measures — "e-indicators."

As part of their efforts, the researchers:

  • Completed a search through the research literature for existing quality performance e-indicators.
  • Examined the experiences of five different integrated health care delivery systems in developing, testing and implementing quality performance e-indicators.

Key Findings

  • The results of the literature search revealed that few previous studies focused specifically on the development of quality indicators for the new electronic environment.
  • The experience of the five health care delivery systems (profiled in a report available online) revealed that:
    • Performance measures become much more clinically relevant when they are based on the electronic health record.
    • Establishing the validity and reliability of data extracted from the electronic health record is a key challenge to the development and use of e-indicators.
    • Measures that translated established quality indicators had the easiest transition into electronic health record implementation.

Funding
The Robert Wood Johnson Foundation (RWJF) supported this project through an unsolicited grant of $154,948 from December 2005 to November 2007.

 See Grant Detail & Contact Information
 Back to the Table of Contents


THE PROBLEM

Traditional sources of health care data — such as insurance claims information, medical records or patient self-report via special surveys — have yielded performance measures that are widely considered acceptable, but not ideal, for quality improvement, according to researchers at Park Nicollet Institute and the Johns Hopkins Bloomberg School of Public Health.

With new health information technology — including electronic health care records, computerized physician order entry, and clinical decision support systems — health care systems have richer data to evaluate the performance of health care providers, the researchers said. As more health care systems employ electronic health records and related health information technology, health care quality improvement initiatives will accelerate.

When this project was proposed in 2005, use of electronic health records was not yet widespread in the United States (20 to 25 percent of the nation's physicians were reportedly using electronic health records, according to a July 2005 New York Times report). Still, several leading health care provider organizations had begun implementing electronic health records and migrating traditional quality indicators derived from measurement sets from the National Quality Forum (NQF) and the Joint Commission, the researchers said.

To realize the potential offered by electronic health records, however, new standard methods for assessing performance through electronic-health-record-derived data must be developed, the research team said. And the reliability and validity of existing quality indicators that have migrated to electronic health record platform should be tested widely.

 Back to the Table of Contents


RWJF STRATEGY

Electronic Health Records

In the vision of their supporters, electronic health records (EHRs) will provide secure, reliable access to patient health information when and where it is needed to support patient care.

Since around 2000, the Robert Wood Johnson Foundation has supported efforts to expand the use of electronic health records. RWJF's interest in EHRs has been as a tool to improve the quality of care, lower costs and improve consumers' engagement in their health care. As RWJF President Risa Lavizzo-Morey stated in 2004:

Despite all our sophisticated scientific and diagnostic technology, health care doesn't have its own integrated IT system…. Today, our medical and health care information is for others to know and for us to wonder about. 'Knowledge is power' and much of the knowledge about us is kept secret from us. Tomorrow, no more secrets; knowledge will be a power tool that is shared freely. We're finally going to know as much about our health care as the system knows so we'll be able to make informed health care decisions for ourselves and our families.

Thus, RWJF has supported several efforts to develop EHRs, standardize their use and expand their adoption. These efforts have included:

  • Electronic Health Record System Function Model. In 2003, with RWJF grant support, Health Level Seven created an electronic health record system, as well as security standards and communications protocols. Health Level Seven created the Electronic Health Record System Functional Model, and several major industry stakeholders adopted the functional model as a standard for their own companies. (See Grant Results on ID# 049161.)
  • Connecting for Health. In 2004, RWJF joined in co-funding Connecting for Health, an existing initiative begun in 2002 by the Markle Foundation. The initiative laid the groundwork for an effective and secure national health information network that would make patients' medical records available when they are needed, to those authorized to access them. Connecting for Health is a collaboration of representatives from government, health care, industry and consumer groups working to establish policy guidelines and technical solutions to pave the way for nationwide electronic health information exchange. This initiative's overarching goal is a secure environment in which personal medical information is available electronically when and where it is needed by patients and the clinicians who care for them. RWJF's contribution to the effort totaled over $1.5 million between 2004 and 2007. (See Grant Results on ID# 056712.)
  • IOM Summit. Also in 2004, RWJF co-sponsored a summit of over 200 national and community health leaders convened by the Institute of Medicine on strategies to improve patient care for five common chronic conditions. The summit's recommendations related to health care communications and technology included the suggestion federal leadership work to accelerate the adoption of electronic health records. (See Grant Results on ID# 046718.)
  • Surveys of EHR usage. With RWJF support, George Washington University and Massachusetts General Hospital published Health Information Technology in the United States, a survey of how doctors and hospitals are using information systems to drive improvements in quality. The survey estimated that about one-quarter of doctors use EHRs to improve how they deliver care to patients. However, fewer than one in 10 are using what experts define as a "fully operational" system. (See news release and report.) A follow-up survey released in 2008 found a 13 percent increase in EHR usage, but no significant association between EHR use and quality of care. (See report.)
  • Personal health records. Project HealthDesign is a $5-million national program designed to support health and information technology pioneers to create a new generation of personal health record (PHR) systems. Project HealthDesign's goal is to design and test a suite of PHR tools and applications that work together to help people achieve their various and specific health goals in an integrated fashion. In this two-phase, 18-month initiative, nine design teams will create personal health applications that address specific health challenges faced by individuals and families. In the subsequent 12-month prototype phase, these personal health applications will be tested in target populations.

Quality Measurement

RWJF has long seen measuring quality as one aspect in a larger strategy to improve the quality of care. Measurement systems provide a mirror to organizations so they can focus on areas to improve. And they provide accountability to the public so that consumers can choose the highest quality care.

RWJF has supported efforts to develop measures of quality:

 Back to the Table of Contents


THE PROJECT

Researchers from Park Nicollet Institute (the research arm of Minneapolis-based Park Nicollet Health Services) collaborated with researchers from Johns Hopkins Bloomberg School of Public Health through a subcontract to conduct preliminary work in developing a standardized set of electronic health record based quality performance measures — "e-indicators."

As part of their efforts, the researchers:

  • Completed a research literature review for existing quality performance e-indicators.
  • Examined the experiences of five different integrated health care delivery systems in developing, testing and implementing quality performance e-indicators.

The Park Nicollet team led the project logistics; Johns Hopkins' researchers led the literature review; the two teams collaborated on interviews and manuscript development.

The research team convened a seven-member national advisory panel — which included experts in evaluation and quality improvement drawn from government, nonprofit and provider organizations — to provide overall project guidance, review manuscripts and offer channels for dissemination. (For a list of panel members, see Appendix 1.)

Literature Review

Using the National Institutes of Health's PubMed and other sources such as Google Scholar, the research team identified 26 scholarly articles on quality indicators that employed data from electronic health records and/or health information technology systems. From these articles, the team extracted 129 e-indicators. (For examples of e-indicators, see Appendix 2.)

This effort also entailed developing a classification scheme for e-indicators of quality and safety, with special reference to ambulatory care. For details on the typology developed under this grant, see Findings.

Case Studies

The research team worked with a consortium of five integrated health care delivery systems, each of which was an early adopter of electronic health record technology, to develop case studies examining the organization's experiences in developing, testing and implementing e-indicators. The sites and the focuses of their work were:

  • HealthPartners (Minneapolis) used the electronic health record to compile blood pressure measurements.
  • Park Nicollet Health Services (Minneapolis) developed a composite measure for care of people with diabetes.
  • Billings Clinic (Billings, Mont.) tested an automatic alert on potential interactions between antibiotics and the anticoagulant warfarin.
  • Kaiser Permanente (Portland, Ore.) used a natural-language processing tool for counseling about tobacco use.
  • Geisinger Health System (Danville, Pa.) explored ways of reconciling information on the health problem list (a structured portion of the electronic health record that documents and communicates key clinical information) with information in the visit note (an unstructured portion of the electronic health record that captures the physician's diagnoses during the ambulatory visit.)

Communications

The research team completed a report outlining the experiences of the five early adopters, (report available online; see details in the Bibliography). In addition, the investigators made eight presentations on the project to professional groups.

Other Funding

The project received additional funding from the Commonwealth Fund ($153,378) and the Agency for Healthcare Research and Quality ($25,000).

 Back to the Table of Contents


FINDINGS

Literature Review

The researchers summarized their findings from the literature review in a report to RWJF:

  • Few studies focused specifically on the development of quality indicators for the new electronic environment. Most resources are instead being used to adapt the previous generation of performance measures — that is, measures developed for the paper system — into electronic health records.

Case Studies

The researchers summarized their findings from the five case studies in a report, Performance Measures Using Electronic Health Records: Five Case Studies, (available online).

  • Performance measures become much more clinically relevant when they are based on the electronic health record.

    For example, Park Nicollet developed a "composite" e-indicator that tracked the percentage of diabetes patients who achieved specific readings for hemoglobin A1c (a measure of blood sugar control), cholesterol and blood pressure as well as healthy behavior (taking aspirin daily and not using tobacco). Composite measures such as this reflect a more complete clinical picture of a person with diabetes than a single component of the measure can. Such measures can also drive improvements in care in three areas: the individual clinician, the site (and department within the site, if appropriate) and the overall care system.
  • Establishing the validity and reliability of data extracted from the electronic health record is a key challenge to the development and use of e-indicators.

    For example, in terms of validity (i.e., the degree to which a measure actually measures what it was designed to measure), many of the case study participants noted the challenge of correctly defining a specific patient population — a critical prerequisite to accurate assessment — using standard classification codes.

    In terms of reliability (i.e., the extent to which the measurements of a test remain consistent over repeated tests), many case study participants pointed out the difficulty of coding data about patients, diagnoses and procedures in any uniform way. Indeed, some noted that it can be challenging just to record information consistently within one patient's electronic health record.
  • Measures that translated established quality indicators had the easiest transition into electronic health record implementation. Measures incorporating or evaluating features specific to health information technology, such as automated alerts and free-text analysis (i.e., analysis of uncoded text), tended to be specialized to particular systems and not so easily incorporated into other systems.

    Nonetheless, most of the participants were confident that the concepts could be adapted to different types of electronic health record systems and that virtually all of the problems encountered were amenable to performance improvement, often made possible by the electronic health record.
  • E-indicators can be grouped into five broad categories:
    • Translational e-indicators: measures that have been translated from existing measurement sets (e.g., HEDIS or National Quality Forum standard measures) for use in health information technology platforms. (HEDIS — the Healthcare Effectiveness Data and Information Set — is a widely used set of performance measures in the managed care industry, developed and maintained by the National Committee for Quality Assurance.)
    • Health information technology-facilitated e-indicators: measures that, while not conceptually limited to health information technology-derived data sources, would not be operationally feasible in settings without health information technology platforms. For example, measuring clinical physiologic outcomes on 100 percent of patients would not be amenable in traditional systems.
    • Health information technology-enabled e-indicators: measures that generally would not be possible outside of the health information technology context. These indicators are linked to unique health information technology capabilities such as clinical decision support systems, biometric devices or Web-based patient portals.
    • Health information technology-system-management e-indicators: measures needed to implement, manage, evaluate and generally improve health information technology systems. They are primarily intended for use by the parent organization.
    • "E-iatrogenesis" e-indicators: measures of patient harm caused at least in part by the application of health information technology. They assess the degree to which unanticipated quality and safety problems arise, whether of human (provider or patient), technical or organizational/system origin.

 Back to the Table of Contents


CONCLUSIONS

  • For novel, clinically relevant quality measurement to develop, there must be an explicit demand, probably from funders such as RWJF. Without such a stimulus, measurement resources will be restricted to translating current measures into the electronic health record environment.

 Back to the Table of Contents


LESSONS LEARNED

  1. Narrowing the scope of projects such as this allows for a sharper focus and potentially more impact, albeit in a more limited realm. The project director noted that the research team reached for a project too broad in scope, encompassing all performance measures for adult care. (Project Director/Fowles)
  2. Bear in mind that the "snowball" approach to identifying key informants — i.e., relying on referrals made from each consecutive interviewee — precludes the possibility of assembling a full panel of interviewees at the beginning of the project. Though useful in identifying important interviewees, the procedure necessarily slows the interview process. (Project Director/Fowles)

 Back to the Table of Contents


AFTER THE GRANT

The researchers worked to complete several manuscripts related to this project. They also received two additional grants from the Commonwealth Fund to continue work in the development of e-indicators:

  • In 2006, the team completed a project designed to develop, test and refine a set of electronic health record-based performance indicators for well-child care. The measures focused on weight management, developmental screening and care coordination for children.
  • In 2008, the team began a study to explore how electronic health records could facilitate care coordination for adults.

 Back to the Table of Contents


GRANT DETAILS & CONTACT INFORMATION

Project

Development of electronic health record-based indicators for ambulatory and integrated health care systems

Grantee

Park Nicollet Institute (Minneapolis,  MN)

  • Amount: $ 154,948
    Dates: December 2005 to October 2007
    ID#:  053729

Contact

Jinnet B. Fowles, Ph.D.
(952) 993-1949
Jinnet.fowles@parknicollet.com
Jonathan P. Weiner, Dr.P.H.
(410) 955-5661
jweiner@jhsph.edu

 Back to the Table of Contents


APPENDICES


Appendix 1

(Current as of the time of the grant; provided by the grantee organization; not verified by RWJF.)

National Advisory Panel

Helen Burstin, M.D., M.P.H.
Director, Center for Primary Care Prevention and Clinical Partnerships
Agency for Healthcare Research and Quality
Rockville, Md.

Sarah Hudson Scholle, Dr.P.H.
Assistant Vice President, Research & Analysis
National Committee for Quality Assurance
Washington, D.C.

Karen Kmetik, Ph.D.
Director, Clinical Performance Evaluation
American Medical Association
Chicago, Ill.

Gordon Mosser, M.D.
Executive Director
Institute for Clinical Systems Improvement
Bloomington, Minn.

William C. Rollow, M.D., M.P.H.
Director, Quality Improvement Group
Office of Clinical Standards and Quality
Centers for Medicare & Medicaid Services
Washington, D.C.

Cary Sennett, M.D.
Senior Vice President for Research and Development
American Board of Internal Medicine
Philadelphia, Pa.

Reva Winkler, M.D., M.P.H.
National Quality Forum
Washington, D.C.


Appendix 2

(Current as of the time of the grant; provided by the grantee organization; not verified by RWJF.)

Examples of E-indicators

Rates of medication data errors in the electronic medical record: Number of correct medication records divided by the total number of medication records.

Medication costs associated with electronic prescribing system with integrated decision support in primary care: Average drug costs per prescription for all prescriptions.

Rates of adverse drug events due to contraindicated drugs: Number of adverse drug events due to contraindicated drug divided by the number of patients who received a contraindicated prescription.

Cardiovascular disease prevention — risk reduction outcome measures: Change in diastolic blood pressure BP from first contact to first monitoring cycle among hypertensive patients.

Screening rate of smoking status: Patients with smoking status documented in the electronic medical records in the first two days of each month divided by the total number of patients seen during the first two days of each month.

Safety event reporting: Number of safety events reported each month.

 Back to the Table of Contents


BIBLIOGRAPHY

(Current as of date of this report; as provided by grantee organization; not verified by RWJF; items not available from RWJF.)

Articles

Weiner JP, Kfuri T, Chan K and Fowles JB. "e-latrogenesis": The Most Critical Unintended Consequence of CPOE and other HIT." Journal of the American Medical Information Association, 14(3): 387–388, 2007. (Letter to the Editor)

Reports

Fowles JB, Weiner JP, Chan KS, Kind EA and Awwad S. Performance Measures Using Electronic Health Records: Five Case Studies. Commonwealth Fund, 2008. Available online.

 Back to the Table of Contents


Report prepared by: Robert Crum
Reviewed by: Richard Camer
Reviewed by: Marian Bass
Program Officer: Laura C. Leviton