The Program Being Evaluated
Achieving Competence Today (ACT) was a program of Partnerships for Quality Education, a national initiative of the Robert Wood Johnson Foundation (RWJF). ACT tested the use of a Web-based curriculum by medical residents and graduate nursing students at more than 20 schools as a means for them to identify problems in patient care quality and collaboratively use systems-based approaches to improving care. RWJF funded ACT from May 2003 through September 2008.
About the Evaluation
In 2005, RWJF funded evaluators Michael Yedidia, Ph.D., and Colleen Gillespie, Ph.D., of Rutgers University to carry out an evaluation of ACT through a post-training survey of ACT II learners, and pre- and post-training study of ACT III learners and leaders.
The evaluators sought answers in four areas:
- How did learners assess the value of participating in the quality improvement process in the context of their training?
- What was the nature of the quality improvement projects conducted as part of ACT training? How were the projects valued by learners and leaders? How did they assess the contribution of ACT to quality improvement at their institutions?
- To what extent did learners collaborate with peers or staff from different disciplines in developing improvement projects? How did learners and leaders assess the quality of the collaborative experience? What were the factors that either facilitated or impeded effective interdisciplinary collaboration?
- What factors appeared to differentiate those health professions education programs prepared to pay $10,000 to continue participation in ACT from those that didn't?
Summary of Methods
To carry out the evaluation, Yedidia and Gillespie conducted:
- A post-training survey of the ACT II cohort of learners to elicit their assessments of the ACT training and their experience with and evaluation of quality improvement projects.
- A survey of leaders of the ACT initiatives to assess their valuation of the learning experience; their appraisal of the contribution of the projects to quality improvement at their institutions; and their estimation of the centrality of interprofessional collaboration to the ACT initiatives.
- A pre-training and post-training survey of ACT III learners and a survey of ACT III leaders (faculty preceptors, medical education or quality improvement directors) to yield data for analyzing determinants of their decisions as to whether or not to continue participation with no funding.
Knowledge and Impact
In a report on ACT submitted to RWJF in January 2007, the evaluation team reported the following:
- Learners (medical residents or graduate nursing students) commended the ACT training overall. Some 80 to 89 percent of ACT learners "strongly agreed" or "somewhat agreed" that they would recommend the initiative to others.
- Among key elements of ACT training, learners most highly rated the development of a quality improvement project. Content areas most frequently chosen by learners for their projects were:
- Effectiveness of care—providing appropriate care to those who could benefit and avoiding unnecessary care.
- Timeliness—reducing patient wait times and potentially harmful delays.
- Patient safety.
- ACT leaders (medical education or quality improvement directors, ACT preceptors and teachers) concurred with learners in their assessment of the value of learners' quality improvement projects to their institutions. More than 88 percent of leaders and learners felt they were of moderate or great value.
- Interdisciplinary collaboration was viewed, generally, as very important to the work of quality improvement, with 65 percent of leaders and 69 percent of learners holding this view. Learners, regardless of discipline, credited their ACT training with improving their outlook on two key dimensions of interdisciplinary work:
- Developing trust among collaborators.
- Becoming more aware of the limits of a single-disciplinary approach to care.
- When asked what most facilitated collaboration, learners' most prominent response was "the necessity of getting the job done."