September 2009

Grant Results

SUMMARY

From 2005 to 2008, researchers at the RAND Corporation spurred the use of quality improvement—a data-driven approach to performance often used in industry, the military and health care, but not public health—to boost emergency preparedness among state and local public health agencies.

The researchers published a white paper and articles on the quality improvement approach, developed a pilot collaborative to help five public health agencies use it to prepare for pandemic flu and created a toolkit to encourage other agencies to use quality improvement to improve emergency preparedness.

Key Findings
Researchers reported the following findings in "Using Quality Improvement Methods To Improve Public Health Emergency Preparedness: PREPARE For Pandemic Influenza" (Health Affairs, 27[5], 2008; online version, July 2008):

  • The quality improvement collaborative provided substantial evidence that such an approach can boost emergency preparedness and overall performance of public health departments.
  • The public health teams participating in the collaborative were enthusiastic about applying quality improvement strategies to emergency preparedness, as well as to the day-to-day activities of public health agencies.

Key Results
Researchers reported the following additional results to the Robert Wood Johnson Foundation (RWJF):

  • RAND published Quality Improvement in Public Health: A Way Forward (2006), a white paper that makes the case for using quality improvement (QI) in public health and suggests how to speed its integration into public health practice.
  • RAND researchers developed and disseminated PREPARE for Pandemic Influenza: A Quality Improvement Toolkit to help state and local health departments incorporate quality improvement strategies into emergency preparedness. The toolkit is available on the RWJF Web site and the RAND Center for Public Health Preparedness Web site.
  • The pilot collaborative "galvanized a QI movement in public health," according to Project Co-Director Nicole Lurie, M.D., M.S.P.H. The collaborative not only raised awareness of the need for quality improvement in public health, but also provided a concrete example by applying the approach to preparedness for pandemic flu.

Funding
RWJF provided $328,322 through three grants from June 2005 to October 2008. The Office of the Assistant Secretary for Preparedness and Response of the U.S. Department of Health and Human Services also provided funding under two larger contracts with RAND for work on pandemic flu and emergency preparedness.

 See Grant Detail & Contact Information
 Back to the Table of Contents


THE PROBLEM

In the wake of September 11th, the threat of anthrax, pandemic flu and other public health crises, the ability of the U.S. public health system to respond to emerging threats has taken on new importance.

Quality improvement—a data-driven process that identifies desired outcomes, measures performance and promotes best practices—had become important in industry, the military and health care. That approach also can help create adaptable, effective public health organizations, according to researchers at the RAND Corporation, a nonprofit research organization in Santa Monica, Calif. However, it was underused in public health, and agencies lacked widely tested models for applying it.

Public Health Preparedness

In 2002, RAND researchers studied nationwide public health preparedness and the ability of local health departments to respond to threats such as epidemics, under a contract with the federal Department of Health and Human Services (DHHS), Office of the Assistant Secretary for Preparedness and Response. The investigators found substantial variation in performance among the agencies: Some responded well, whereas others performed poorly.

The researchers also found that:

  • Standard definitions and measures of preparedness were lacking, along with measures to assess the performance and progress of health departments.
  • Reports on the outcomes of public health preparedness drills and exercises often sat unused on shelves.
  • Agencies have made few systematic efforts to close gaps in preparedness after identifying them.

The RAND researchers participated in an RWJF meeting on applying the quality improvement approach to public health on November 10, 2004. After talking with staff members from the Institute for Healthcare Improvement (IHI), Cambridge, Mass., at the meeting, the RAND researchers decided to work with them to promote the use of quality improvement methods in public health.

IHI has been the national program office for three RWJF programs:

 Back to the Table of Contents


RWJF STRATEGY

RWJF has been involved in quality improvement efforts in health care for many years. Funding has covered numerous areas of the health care system, including:

The Public Health Team has built upon these efforts in developing its strategy to improve performance in public health. The team's Strategy Statement on the Foundation's Web site states:

"For the public health system to fulfill its important role in ensuring the safety and health of the public, we must drive fundamental improvements in the quality, performance and impact of public health agencies. We are advancing efforts to help public health agencies improve the services they provide and increase accountability to the communities they serve. We support efforts to establish a national accreditation system for state and local public health agencies. Accreditation will establish agency standards and benchmarks that promote excellence, continuous quality improvement and accountability for the public's health.

"Quality improvement efforts are not limited to public health agencies. We seek to improve the performance within and across the public health system. To that end, we foster collaboration among federal, state and local public health agencies and others integral to the public health system, such as businesses, health care providers, educational institutions, and faith- and community-based organizations. Recognizing the importance of strong leaders to strengthen the system, we are supporting a nationwide leadership program designed to improve cooperation among federal, state and local leaders from the private, public and not-for-profit sectors to work collaboratively to increase preparedness for the challenges of both manmade and natural disasters."

 Back to the Table of Contents


THE PROJECT

Researchers at the RAND Corporation spurred the use of quality improvement—a data-driven approach to performance often used in industry, the military and health care, but not public health—to boost emergency preparedness among state and local public health agencies, under three grants from RWJF.

Making the Case for Quality Improvement

During the first grant (ID# 053270; June 2005 to December 2005), RAND researchers worked with staff at the Institute for Healthcare Improvement (IHI) to create a white paper showing how to apply quality improvement in public health. (For information on the paper's contents, see Results and the Appendix.)

To identify best practices, the researchers conducted seven site visits and two telephone interviews with nine exemplary state and local health departments that were known to have made big strides in improving their emergency preparedness, or to be advancing quality improvement and accreditation efforts. They also developed a follow-up plan calling for a pilot learning collaborative to apply quality improvement to emergency preparedness in public health—specifically a global outbreak of influenza or pandemic flu.

Developing the Learning Collaborative

The RAND researchers began planning the pilot collaborative—called Promoting Emergency Preparedness and Readiness for Pandemic Influenza, or PREPARE for PI—in 2005, under a contract with the Office of the Assistant Secretary for Preparedness and Response. The collaborative was part of RAND's work on pandemic flu preparedness under a $2-million contract with the federal Department of Health and Human Services (DHHS).

Although IHI had extensive experience with quality improvement collaboratives, it relied on a model targeted to chronic care and clinical practice rather than public health. RAND researchers therefore began working with staff at the Center for Health Care Quality at Cincinnati Children's Hospital and Medical Center, which had participated in learning collaboratives in public health.

Because PREPARE for PI was the first quality improvement collaborative in emergency preparedness, and one of only a few in public health, the researchers realized that they needed more time to develop a framework and performance measures and to educate participants. RWJF therefore provided funding to extend the time period for the collaborative (Grant ID# 058603; November 2006 to June 2007).

The Collaborative Framework
To develop a framework for the collaborative's work, RAND researchers convened a team of experts in emergency preparedness and quality improvement from RAND and Cincinnati Children's Hospital and Medical Center. The resulting framework included four components:

  • Setting aims and goals.
  • Creating performance measures.
  • Devising strategies and ideas for change in preparedness.
  • Using rapid plan-do-study-act cycles: that is, planning a change, trying it out on a small scale, measuring the impact and using the results to inform the next improvement cycle.

The planners agreed to apply this framework to five activities designed to reduce mortality, morbidity and social disruption in the event of a flu pandemic:

  • Surveillance
  • Case investigation
  • Command and control
  • Risk communication
  • Disease control and treatment

The Quality Improvement Process
Three- or four-person teams from five health departments—two state and three local—agreed to participate in the collaborative to improve their pandemic flu preparedness, from May 2006 to February 2007. The departments were:

  • Baltimore City Health Department
  • Genesee County Health Department (Michigan)
  • Georgia Division of Public Health
  • Multnomah County Health Department (Oregon)
  • Virginia Department of Health

The teams attended three 1.5-day learning sessions (May and September 2006 in Washington, and February 2007 in Santa Monica, Calif.). The teams also submitted monthly progress reports to RAND and participated in conference calls with project staff and other experts.

During the learning sessions, the teams chose which of the five key activities they would focus on and created goals and performance measures for each activity. The teams then used small, rapid-cycle tests to implement changes in preparedness at their agencies. RAND researchers evaluated the results through an online survey, interviews with team members and the monthly team reports. (See Findings for more information.)

Sharing Results and Tools

To encourage other public health departments to integrate quality improvement into their work, RWJF then funded RAND to create and disseminate results and tools from the learning collaborative (Grant ID# 062347; August 2007 to December 2008). To develop a toolkit, project staff convened an advisory board of experts in public health and the use of quality improvement and also solicited input from a RAND advisory board.

The Office of the Assistant Secretary for Preparedness and Response at DHHS provided funding to enable project staff to search the Web for best practices in emergency preparedness for use in the toolkit. This effort was part of RAND's work on pandemic flu preparedness under a $2-million contract with DHHS.

Communications

Project staff produced a white paper, articles, a toolkit and a webinar (online seminar) on using quality improvement in public health preparedness and disseminated these products through the Web and at national conferences and other events. See Results and the Bibliography for details.

 Back to the Table of Contents


FINDINGS

RAND researchers reported the following findings from studying the collaborative in "Using Quality Improvement Methods To Improve Public Health Emergency Preparedness: PREPARE For Pandemic Influenza" (Health Affairs, July 2008, online issue):

Improved Agency Preparedness

  • The Promoting Emergency Preparedness and Readiness for Pandemic Influenza (PREPARE for PI) experience provided substantial evidence that a quality improvement collaborative can boost emergency preparedness and overall performance of public health departments. Teams moved from planning to implementing many strategies critical to preparing for pandemic flu and other public health emergencies. For example:
    • The Georgia Division of Public Health used data from small tests to obtain realistic estimates of staffing and resource needs for a new triage telephone line. This plan-do-study-act process also enabled the quality improvement team to win buy-in from nursing staff despite earlier resistance.
    • Oregon's Multnomah County Health Department used plan-do-study-act cycles to improve its ability to respond to emergency requests from hospitals for medical supplies and volunteers during a pandemic emergency. The cycles also helped clarify the roles and responsibilities of the public health agency and its hospital partners.
  • The teams were enthusiastic about applying quality improvement strategies to emergency preparedness as well as to the day-to-day activities of public health agencies:
    • The process of breaking broad preparedness concepts into pieces and conducting small tests helped teams develop and test quality improvement at little cost.
    • Establishing baseline performance helped teams set goals for quality improvement and measure progress.
    • Teams built preparedness by improving on measures of performance, strengthening relationships with internal or external partners or both.
    Several teams used these methods after the collaborative ended and planned to continue to do so.
  • The majority of participants believed that the pilot learning collaborative was valuable. Participants:
    • Relied on team members from other health departments for new ideas and support in making needed changes in preparedness.
    • Found spending time with experts during conference calls and learning sessions especially helpful.
    • Improved their understanding of colleagues' roles and learned how to make better use of their department's resources.

Policy Implications

  • Clarify public health processes and develop reliable performance measures. The public health community, academia and government should work together to outline these processes and measures.
  • Create the right incentives. Both financial and nonfinancial incentives can help ensure that public health agencies are accountable and commit to continuous improvement. For example, federal and state governments could link funding for emergency preparedness to improved performance and the use of quality improvement practices. Public recognition for high-performing agencies could encourage officials to use that approach.
  • Develop public health expertise in quality improvement. "Learning organizations" and trainers who understand public health could disseminate quality improvement skills and techniques within the public health community.
  • Demonstrate and evaluate larger-scale quality improvement efforts to show how the approach can apply to public health.

 Back to the Table of Contents


RESULTS

Researchers reported the following additional results to RWJF:

  • RAND published Quality Improvement in Public Health: A Way Forward (2006), a white paper that makes the case for using quality improvement (QI) in public health and suggests how to speed its integration into public health practice. See the Appendix for more on the factors that contribute to the QI gap in public health, and recommendations on how to close it.

    RAND researchers published "Quality Improvement in Public Health Emergency Preparedness," an article in the Annual Review of Public Health (28, April 2007).
  • RAND researchers developed and disseminated PREPARE for Pandemic Influenza: A Quality Improvement Toolkit (2008) to help state and local health departments incorporate quality improvement strategies into emergency preparedness.

    The toolkit—available on the RWJF Web site and the RAND Center for Public Health Preparedness Web site—shows how to use quality improvement strategies in six key areas:
    • Surveillance
    • Case reporting and investigation
    • Command and control
    • Risk communication
    • Disease control
    • Disease treatment
    For more information see the Web site for the RWJF-funded national program Lead States in Public Health Quality Improvement.
  • Researchers raised awareness of the toolkit and white paper through a webinar and extensive outreach to the public health community.

    The Webinar
    "Quality Improvement Strategies for Public Health Emergency Preparedness"—held on November 19, 2008, the month before the release of the toolkit—drew 265 registrants and 150 participants (the maximum number the event could accommodate).

    Participants represented nearly all states and many national groups, including the Centers for Disease Control and Prevention (CDC) and the Department of Health and Human Services (DHHS). The webinar:
    • Showed how agencies can use quality improvement methods to identify areas for improvement and to develop strategies for making effective changes.
    • Provided examples of firsthand experiences from the pilot quality improvement learning collaborative in public health emergency preparedness.
    • Showed how to use small-scale drills to assess performance and make continuous improvements.
    The webinar is available online. Also see the Bibliography.

    Outreach
    RAND researchers invited 2,300 members of the public health community to obtain the toolkit and participate in the webinar. Project staff distributed 500 CD-ROMs of the toolkit at the 2009 Public Health Preparedness Summit and to participants in RWJF's Lead States in Public Health Quality Improvement (originally called the Multi-State Learning Collaborative).

    Project staff also notified a listserv of the National Association of County and City Health Officials about the webinar and toolkit. People who found the toolkit online could request the CD-ROM version.

    Researchers also made presentations on the project in 2007 and 2008, including at meetings of the following:
    • Association of State and Territorial Health Officials
    • Robert Wood Johnson Clinical Scholars Program
    • Lead States in Public Health Quality Improvement
    • National Association of County and City Health Officials' Conference
    • Public Health Preparedness Summit
    • AcademyHealth
  • RWJF named "Using Quality Improvement Methods To Improve Public Health Emergency Preparedness: PREPARE For Pandemic Influenza" (Health Affairs, July 2008) one of the 10 most influential RWJF-funded research articles of 2008.

    David C. Colby, Ph.D., RWJF vice president for research and evaluation, selected the article because it "highlights how QI methods are feasible, cost-effective, and sustainable by small phased-in changes that improve regular work instead of adding an additional burden in public health settings."
  • The pilot collaborative "galvanized a QI movement in public health," according to Project Co-Director Nicole Lurie. The collaborative not only raised awareness of the need for quality improvement in public health, but also provided a concrete example by applying the approach to preparedness for pandemic flu.

    For example:
    • The National Association of County and City Health Officials—now headed by a participant in the collaborative—has begun promoting the use of quality improvement within local health departments.
    • DHHS has hired its first quality improvement staff and is using the approach to develop a national health security strategy.
    • A former participant in the collaborative who is now deputy director of the Food and Drug Administration is using quality improvement to plan for emergencies.

 Back to the Table of Contents


LESSONS LEARNED

  1. Convincing the public health community to accept a tool used in health care is difficult. Some public health professionals fear the "medicalization" of public health, and that creates barriers to using quality improvement. Project Co-Director Nicole Lurie recommends continuing to educate public health practitioners about the benefits of the approach.
  2. The public health community may have little understanding of quality improvement concepts. Because public health agencies "typically deal with large numbers," according to Lurie, the Promoting Emergency Preparedness and Readiness for Pandemic Influenza (PREPARE for PI) teams did not understand the benefit of making small, incremental changes or how quality improvement could work with existing systems. Building trust and creating language and examples that team members could relate to took more time than expected.
  3. Start quality improvement efforts with a few targeted, clearly outlined public health processes. Teams found it difficult to work simultaneously in multiple domains of preparedness, such as surveillance, risk communication and disease control and treatment. Choosing the domains of greatest interest to the teams took time. (Project Co-Director/Lurie)
  4. Begin quality improvement efforts with reliable, easily tested performance measures. The teams spent a great deal of time struggling to develop performance measures and required extensive assistance from project staff. (Project Co-Director/Lurie)
  5. Public health teams can learn quality improvement by applying it to their own needs. The teams appreciated the opportunity to learn quality improvement methods by tackling an area they were committed to improving. (Project Co-Director/Lurie)
  6. Use webinars to communicate with practitioners nationwide. The toolkit webinar was an efficient and effective way to reach widely dispersed public health departments and practitioners, especially in a time of tight budgets. However, creating a webinar and products such as the toolkit requires partners who know how to tailor them to a broad audience. (Project Co-Director/Lurie)

 Back to the Table of Contents


AFTER THE GRANT

As of May 2009, RAND is continuing to promote the use of quality improvement methods and tools in public health and emergency preparedness. For example, the organization is working with the Centers for Disease Control and Prevention to develop accountability measures for state and local public health emergency preparedness.

The project has had a significant impact on the way RAND disseminates its work on emergency preparedness, according to the project co-directors. For example, the organization is now creating more user-friendly CD-based toolkits and other products for use by the public health community and also is offering more webinars to such practitioners.

 Back to the Table of Contents


GRANT DETAILS & CONTACT INFORMATION

Project

Using quality improvement methods to improve public health emergency preparedness

Grantee

RAND Corporation (Arlington,  VA)

  • Planning for the development of a systems-based strategy to improve the quality of public health
    Amount: $ 94,925
    Dates: June 2005 to December 2005
    ID#:  053270

Contact

Contact: Nicole Lurie, M.D., M.S.P.H.
(703) 413-1100 ext. 5127
lurie@rand.org

Grantee

RAND Corporation (Santa Monica,  CA)

  • Extending a learning collaborative for public health/pandemic preparedness
    Amount: $ 133,469
    Dates: November 2006 to June 2007
    ID#:  058603

  • Dissemination of the results and tools from a learning collaborative for quality improvement in public health and pandemic influenza preparedness
    Amount: $ 99,928
    Dates: August 2007 to October 2008
    ID#:  062347

Contact

Contact: Debra Lotstein, M.D., M.P.H.
(310) 393-0411, ext. 6076
lotstein@rand.org

 Back to the Table of Contents


APPENDICES


Appendix 1

Findings and Recommendations in Quality Improvement in Public Health: A Way Forward (2006)

Factors Contributing to the Quality Improvement Gap Between Public Health and Other Fields

  • Quality improvement methods were designed for "production systems" that are well defined and controlled. Public health is far more variable and entails multiple agencies, which diffuses responsibility and accountability.
  • Work processes in public health have not been well defined, making it difficult to identify specific levers of change.
  • Because public health outcomes focus on prevention or response to rare events, officials cannot measure those outcomes as easily as medication errors in hospitals.
  • Public health agencies lack key resources for quality improvement:
    • Leaders trained in the process
    • Information systems for tracking performance
    • The expertise to develop improvement plans
  • There is no market for quality improvement in public health, and no "purchaser" who can hold agencies accountable for lack of quality or reward them for improvement.
  • In a time of tight budgets, quality improvement competes with other public health priorities for time and funding.

Recommendations

  • Communicate the need. Building a strong case that uneven performance in the public health system puts people at risk could spur the public and policy-makers to action.
  • Build basic quality improvement abilities and capacity. Tools for doing so include development grants, education and training, technical assistance, tool development and grants that reward quality improvement practices. (Note from the project director: National public health accreditation, led by the Public Health Accreditation Board, is expected to launch in 2011 and to foster a culture of quality improvement in public health departments.)
  • Create the right incentives to encourage quality improvement efforts in public health. Financial incentives and public recognition are two possible options.
  • Clarify public health processes and develop reliable measures. Developing these measures should be the shared responsibility of the public health community, academia and government.
  • Provide leadership and skills. Few quality improvement consultants have expertise in public health.
  • Capitalize on collaboration. RAND's contract with the U.S. Department of Health and Human Services to develop a collaborative for pandemic flu preparedness is an example.

 Back to the Table of Contents


BIBLIOGRAPHY

(Current as of date of this report; as provided by grantee organization; not verified by RWJF; items not available from RWJF.)

Articles

Lotstein D, Seid M, Ricci K, Leuschner K, Margolis P and Lurie N. "Using Quality Improvement Methods To Improve Public Health Emergency Preparedness: PREPARE For Pandemic Influenza," Health Affairs, 27(5); Web Exclusive: w328–w339, July 2008. Abstract available online.

Seid M, Lotstein D, Williams VL, Nelson C, Leuschner KJ, Diamant A, Stern S, Wasserman J and Lurie N. "Quality Improvement in Public Health Emergency Preparedness," Annual Review of Public Health, 28: 19–31, 2007. Abstract available online.

Reports

Seid M, Lotstein D, Nelson C and Lurie N. Quality Improvement in Public Health: A Way Forward. Arlington, VA: RAND, 2006.

Toolkits

Lotstein D, Leuschner KJ, Ricci KA, Ringel JS and Lurie N. PREPARE for Pandemic Influenza: A Quality Improvement Toolkit. Santa Monica, CA: RAND, 2008. Available on both rwjf.org and rand.org.

Webinars

"Quality Improvement Strategies for Public Health Emergency Preparedness," held November 19, 2008. Available online.

 Back to the Table of Contents


Report prepared by: Lori De Milto
Reviewed by: Sandra Hackman
Reviewed by: Molly McKaughan
Program Officer: Susan B. Hassmiller
Program Officer: Terry L. Bazzarre
Program Officer: Abbey K. Cofsky

Most Requested