September 2010

Grant Results

SUMMARY

During 2006 and 2007, researchers at the National Board of Medical Examiners and the Department of Family Medicine at the Robert Wood Johnson Medical School laid the groundwork for an early warning system to identify physicians at risk for performance problems and detect and remedy deficits in physician competence and performance early enough to reduce the potential for patient harm. Researchers:

  • Characterized the types of physician problems encountered by state medical licensing boards and other organizations involved in physician education and assessment.
  • Developed an understanding of the physician assessment and remedial intervention needs of the licensing boards and other stakeholders, especially their degree of acceptance of a set of six core competencies as an organizing framework for physician performance assessment.

Key Findings

Data from the Board Action Data Bank of the Federation of State Medical Boards as well as surveys of state medical board members showed that the most frequently received complaint about physicians related to "conforming to minimal standards of acceptable medical practice."

  • A survey of physicians identified "assisting patients in dealing with health system complexities" as needing the most improvement among their peers.
  • Despite much agreement about the importance of the individual areas covered by the six competencies, state medical boards and other stakeholders do not universally accept the set of six competencies as an organizing framework.
  • Health care system issues and physician performance are often interrelated, and it is difficult for state medical boards to separate out practitioner issues from hospital or health care system issues.

Funding

The Robert Wood Johnson Foundation (RWJF) supported this project through an unsolicited grant of $219,770.

 See Grant Detail & Contact Information
 Back to the Table of Contents


The Problem

According to a report released by the Federation of State Medical Boards, between 2000 and 2004 the number of disciplinary actions taken by state medical boards against physicians rose almost 36 percent, from 4,617 to 6,265 (Federation of State Medical Boards Summary of 2004 Board Actions; available online.)

The subset of physicians with quality problems that are brought to the attention of medical boards and other accrediting or credentialing organizations may represent the "tip of the iceberg" of problem physicians encountered by the public, as the nation lacks adequate measures for an early assessment of physician performance problems (Wainer H and Mee J. Evaluation & the Health Professions, 27(4): 369–382, 2004; abstract available online).

State medical boards lack adequate ways to proactively identify physicians who may have performance problems. Physicians in the United States are licensed to practice medicine by state medical boards after successfully completing the U.S. Medical Licensing Exam. For some 18 percent of physicians who do not pursue specialty board certification, initial licensure is the only time during their career at which their knowledge and competence are formally assessed, according to researchers at the National Board of Medical Examiners.

According to the National Board of Medical Examiners, a more satisfactory system would encourage continuous, publicly verifiable physician assessment beginning at the onset of graduate training and continuing throughout the physician's practicing lifetime of patient care. Such a system would reward physicians for identifying and correcting potential deficiencies in their practice before problems affect the quality of patient care. This would create, in effect, an early warning system for physician performance problems.

In order to develop such a system, a first step is to define and characterize the problems currently seen in practicing physicians in the United States.

 Back to the Table of Contents


RWJF Strategy

According to RWJF's 2005 Annual Report, our immediate focus in the Pioneer Portfolio in 2005 was "to develop a strategic approach that will attract new ideas and breakthrough innovations. Our longer-term objective is to assess and steer our investments toward projects that are most likely to result in breakthrough advances in health and health care."

 Back to the Table of Contents


The Project

The National Board of Medical Examiners, based in Philadelphia, is an independent, not-for-profit organization responsible for preparing and administering qualifying examinations for physicians and other health professionals. In this project, researchers at the National Board of Medical Examiners laid the groundwork for an early warning system to identify physicians at risk for performance problems by:

  • Describing and characterizing the types of problems state medical licensing boards, hospital credentials committees, insurers, medical group practices and other health care related entities encounter among licensed physicians in the United States.
  • Developing an understanding of the needs of the groups listed above for addressing problems in physician assessment and remedial intervention.

Of particular interest to the research team was the degree to which state medical boards and other stakeholders accepted a set of six specific core competencies as an organizing framework for the assessment of physician performance. These are:

  • Patient care.
  • Medical knowledge.
  • Practice-based learning and improvement.
  • Interpersonal and communication skills.
  • Professionalism.
  • Systems-based practices.

Endorsed in 1999 by the Accreditation Council for Graduate Medical Education, the six competencies are notable for putting interpersonal, communication and professional skills on a par with patient care and medical knowledge. For more details on the six competencies, see Appendix 1.

The research team used a combination of quantitative and qualitative data collection strategies for this project. For assistance with the qualitative assessment and analysis, the team subcontracted with the Research Division of the Department of Family Medicine at the Robert Wood Johnson Medical School.

Quantitative Data Collection

  • Researchers reviewed and analyzed the Board Action Data Bank of the Federation of State Medical Boards to determine the most frequent reasons member boards took action against physicians in the calendar year 2006.
  • Researchers conducted:
    • A survey of 28 of the 66 total state and territory (e.g., Puerto Rico, Guam, Virgin Islands, etc.) medical boards (including separate boards of osteopathic medicine in some states) about:
      • The frequency of complaints about physicians.
      • Board member perceptions regarding these complaints.
    • A survey of 717 physicians to determine:
      • How often they had the opportunity to observe or interact with other physicians in matters concerning patient care.
      • Their perspectives on which competencies needed the most improvement among physicians they observed.
    • An Internet survey of 406 pharmacists on their experiences with:
      • Physician prescribing practices.
      • Collaboration with physicians as part of the health care team.

Qualitative Data Collection

  • Researchers conducted three focus groups, with five to 12 attendees per group, with participants in the 2006 annual meeting of the Federation of State Medical Boards.
  • To further understand issues raised by the quantitative data analysis and focus groups, researchers conducted interviews with:
    • Board members and staff during site visits to three state medical boards.
    • Some 40 leaders of national organizations, such as the Accreditation Council for Graduate Medical Education and the American Board of Medical Specialties.

See Appendix 2 for details on the quantitative and qualitative data collection.

Communications

During the course of the project, the research team made five presentations, including one poster presentation, at professional conferences, including the Coalition for Physician Enhancement and the North American Primary Care Research Group.

 Back to the Table of Contents


Findings

Researchers reported findings in an unpublished paper and in a report to RWJF; they reported findings from the survey of pharmacists only in a report to RWJF.

  • State medical board survey respondents reported that the most frequently received complaint about physicians related to "conforming to minimal standards of acceptable medical practice."
    • Respondents also considered this to be the complaint of greatest concern.
    • In addition, it was the most frequently appearing item in the Board Action Data Bank of the Federation of State Medical Boards.
  • State medical board respondents indicated that the areas with the most significant need for assessment were:
    • "Having an adequate fund of knowledge" (cited by 80 percent).
    • "Exercising good clinical judgment" (cited by 78 percent).
  • Overall, surveyed physicians identified "assisting patients in dealing with health system complexities" as needing the most improvement among their peers. However, while 50 percent of internists indicated a need for improvement on this, only 28 percent of family physicians and obstetrician/gynecologists agreed.
  • Surveyed pharmacists reported that they contacted physicians at least several times per week regarding:
    • "Selecting strength [of a prescribed medication] or doing [dosing] schedule" (55 percent).
    • "Screening for potential drug interactions" (54 percent).
    • "Choice of [prescribed] medication" (46 percent).
  • Surveyed pharmacists indicated that physicians needed significant improvement in:
    • "Practicing cost-effective care in prescribing" (63 percent).
    • "Prescribing within the restrictions of patient's insurance" (62 percent).
  • While members of state medical boards and other stakeholders generally agree about the importance of the individual areas covered by the six competencies, they do not universally accept the set of competencies as an organizing framework. In fact, each state medical board interviewed used its own model to organize consumer complaints.
  • Although many problems seen by state medical boards are at the level of individual practicing physicians, physician behavior is often interrelated with the health care system in which the physician practices and how that system operates. For instance, interviewees felt that, while problems at the hospital or health care institution level may have the potential for adverse effects on quality of care and patient safety, current physician assessment systems are designed only to look at individual behaviors, independent of the role the larger system may have. Such situations are particularly difficult for state medical boards and other entities to address.
  • A significant problem area for practicing physicians, and often the root cause of other problems, is interpersonal and communication skills, according to state board personnel and other stakeholders. Interviewees felt that opportunities for enhancing communication skills were inadequate in medical education. Many also identified health care system issues and the increasing complexity of patient care as underlying causes of complaints about communication.
  • The current organization of competency assessment and maintenance is mostly punitive and contributes to a culture that does not enable self-assessment, lifelong learning or self-remediation efforts. Many interviewees, especially board personnel, complained about the nonexistence of early warning systems. In addition, interviewees did not view the existing continuing medical education system as supportive in this regard.

 Back to the Table of Contents


Conclusions

Researchers offered the following conclusions in an unpublished paper:

  • A critical first step in developing reliable assessment methods is "to delineate a consistent national framework for defining physician competence and performance objectives, and to develop a coherent and defensible approach to assessment and continuing education of practicing physicians.

    "Such an approach should support continued improvement of the physician workforce, while also including a mechanism for removing from practice physicians who pose significant risks to patient safety and welfare. Current regulatory practices seem to fall short in both regards."
  • In addition to developing effective assessment methods for individual physicians, it is also important "to address health care systems problems that prohibit improvement in physician performance and health care quality."

 Back to the Table of Contents


Lessons Learned

  1. Use a range of data collection methods to deepen and enrich the findings. In this project, the research team not only combined quantitative and qualitative research methods, they also continuously synthesized and summarized the qualitative data as it was collected. As the data expanded, the team gleaned deeper insights into the data and was able to tap new sources for additional data and information and further refine the data collection instruments. (Project Director/Hawkins)
  2. Contact key stakeholders prior to the commencement of the study to enhance the collaborative nature of the project. Because of the sensitive nature of this project—both politically and in terms of confidentiality—the research team had to contact stakeholders before the project actually got underway. The advance contact redounded to their benefit through enhanced collaboration. (Project Director/Hawkins)

 Back to the Table of Contents


Afterward

As of November 2008, the research team was putting together a task force—composed of representatives of licensing, assessment and physician groups—to establish priorities in research, infrastructure, assessment and education that are necessary to support the development of effective ongoing assessment methods for physicians.

The findings from this project, gathered in an article the research team plans to publish, will inform the deliberations of the task force. The group will be sponsored by the Post Licensure Assessment System, a program that was established in 1998 by the National Board of Medical Examiners and the Federation of State Medical Boards to assist medical licensing authorities in assessing physicians who have already been licensed.

 Back to the Table of Contents


GRANT DETAILS & CONTACT INFORMATION

Project

Laying the Groundwork and Providing a Framework for an Early Warning System to Identify Physicians at Risk for Performance Issues

Grantee

National Board of Medical Examiners (Philadelphia,  PA)

  • Amount: $ 219,770
    Dates: January 2006 to June 2007
    ID#:  055482

Contact

Richard E. Hawkins, M.D.
(215) 590-9500
rhawkins@nbme.org

 Back to the Table of Contents


APPENDICES


Appendix 1

(Current as of the time of the grant; provided by the grantee organization; not verified by RWJF.)

The Six Core Competencies Endorsed by the Council for Graduate Medical Education

In 1999, the Accreditation Council for Graduate Medical Education endorsed six core competencies as areas that residency programs must require their residents to obtain. The council defined the competencies as follows (available online):

  • "Patient Care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health."
  • "Medical Knowledge about established and evolving biomedical, clinical, and cognate (e.g. epidemiological and social-behavioral) sciences and the application of this knowledge to patient care."
  • "Practice-Based Learning and Improvement that involves investigation and evaluation of their own patient care, appraisal and assimilation of scientific evidence, and improvements in patient care."
  • "Interpersonal and Communication Skills that result in effective information exchange and teaming with patients, their families, and other health professionals."
  • "Professionalism, as manifested through a commitment to carrying out professional responsibilities, adherence to ethical principles, and sensitivity to a diverse patient population."
  • "Systems-Based Practice, as manifested by actions that demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value."


Appendix 2

(Current as of the time of the grant; provided by the grantee organization; not verified by RWJF.)

Data Collection Details

Quantitative Data Collection

  • Researchers reviewed and analyzed the Board Action Data Bank of the Federation of State Medical Boards to determine the most frequent reasons actions were taken against physicians across all member boards in the calendar year 2006.
  • Researchers conducted a survey of state medical boards to gather supplemental information about the frequency of complaints received about physicians by the boards, and to elicit board member perceptions regarding these complaints. Of the e-mail invitations sent to the 66 boards, 28 (45 percent) resulted in completed surveys.
  • Researchers conducted a survey of 717 physicians to determine how often they had the opportunity to observe or interact with other physicians in matters concerning patient care, and then to collect responses to a series of items directly relating to the Accreditation Council for Graduate Medical Education core competencies. Researchers administered the questionnaire during 2006 at three professional conferences:
  • Researchers conducted an Internet survey of 406 pharmacists—recruited through the American Association of Colleges of Pharmacy Web site and by invitation in the organization's newsletter—to gather information about their experiences with physician prescribing practices and collaboration with physicians as part of the health care team. Questions focused on:
    • How frequently they contacted physicians concerning the clinical appropriateness of different elements of prescriptions.
    • Aspects of prescribing most in need of improvement.
    • The amount of improvement needed in various aspects of interaction and communication with physicians.

Qualitative Data Collection

 Back to the Table of Contents


BIBLIOGRAPHY

(Current as of date of this report; as provided by grantee organization; not verified by RWJF; items not available from RWJF.)

Articles

Hawkins R, Roemheld-Hamm B, Ciccone A, Mee J and Tallia A. "A Multimethod Study of Needs for Physician Assessment: Implications for Education and Regulation." Journal of Continuing Education in the Health Professions, 29(4): 220–234, 2009. Abstract available online.

Survey Instruments

"Survey of State Medical Boards," National Board of Medical Examiners, fielded 2006.

"Physician Survey," National Board of Medical Examiners, fielded 2006.

"Pharmacist Survey," National Board of Medical Examiners, fielded 2006.

Presentations and Testimony

Beatrix Roemheld-Hamm, Alfred Tallia, Maria Pellerano, Richard Hawkins, Andrea Ciccone and Janet Mee. "PS 181 National Needs Assessment: Problem Physician Early Warning System." 35th Annual Meeting of the North American Primary Care Research Group, Vancouver, BC, October 21–23, 2007. Abstract available online. Enter first author to see abstract.

Richard Hawkins, Janet Mee, Andrea Ciccone, Al Tallia and Beatrix Hamm. "Doctors' Perceptions Regarding Their Colleagues' Clinical Competence." 13th Ottawa International Conference on Clinical Competence, Melbourne, Australia, March 5–8, 2008. (Page 162 online.)

Richard Hawkins, Janet Mee, Andrea Ciccone, Al Tallia and Beatrix Hamm. "Doctors' Prescribing Practices and Communication with Pharmacists: Opportunities for Professional Development." 13th Ottawa International Conference on Clinical Competence, Melbourne, Australia, March 5–8, 2008. (Page 161 online.)

 Back to the Table of Contents


Report prepared by: Robert Crum
Reviewed by: Mary B. Geisz
Reviewed by: Marian Bass
Program Officer: Rosemary Gibson

Most Requested