ACGME In Practice 2

Table 1: The guiding principles for tool development for the ACGME competencies

1. Use multiple tools. The Task Force recommends against using a single tool (e.g., global evaluation form) to measure all of the ACGME competencies. We recommend that each ACGME competency be assessed by at least three different tools at different times and in different settings by different evaluators. The results of the tools can thus be compared to determine if consistent and similar results are being obtained for the competency in question (i.e., comparative validity).

2. Conduct initial face validity evaluation. We use the Task Force as a “test market” for the proposed tool. A specific tool should measure “what we think it is measuring” and consensus from the Task Force insures face validity prior to implementation. Specific tools (e.g., cataract surgical assessment) may require multi-center face validity testing by circulating the proposed tool to experts in cataract surgery at the home or other institutions.

3. Measure reliability and determine concurrent and discriminative validity. The Task Force reviews the results of each tool in aggregate. Results should be reproducible and should demonstrate inter-rater and intra-rater reliability. A tool must be reliable in order to be valid. By employing multiple tools using multiple observers and over multiple encounters the sample size for assessment can be increased in order to insure sufficient numbers to determine the reliability and reproducibility of the results. One of the easy ways to assess validity for a tool is concurrent and discriminative validity. For example, the summative score of a particular assessment tool for each year of residency (e.g. post-graduate year (PGY) 2 compared to PGY 3 and PGY 4) can be assessed and compared to one another to determine concurrent and discriminative validity. Different tools assessing the same ACGME competency and at the same time in training should produce comparable results (i.e., concurrent validity). In addition, assessment of a different competency but using the same tool at different levels of training can provide evidence for discriminative validity (i.e., PGY 4 should perform better than PGY 2 on same assessment).

4. Insure practicality. All of the Task Force recommended tools should be easily applicable in the clinic or operating room setting, are feasible, portable, and generalizable. We favor tools that can be “piggy backed” conveniently onto existing teaching opportunities. The goal for each assessment tool is to “teach and assess” in the same encounter.

5. Limit the faculty time commitment. All Task Force approved tools undergo an assessment of faculty time burden. In general tools should be self-explanatory, easy to use, require one hour or less time per rotation for the individual faculty member and be inexpensive to administer, maintain, and document.

6. Stay within budget. The tools had to be inexpensive to develop, implement, and maintain with a specific annual budget proposal if there were any costs in materials, manpower, or services beyond existing faculty in-kind rotation resources.

7. Measure useful outcomes data. All Task Force approved tools are expected to produce meaningful quantitative data for linkage to educational outcome assessments for improvement over time in the educational program in the future

8. Insure fairness in the process. The Task Force tools prior to implementation are presented to the residents for review and discussion (usually at our ophthalmology grand rounds). Explicit and public learning objectives, public posting of the scoring rubric and an open and clearly defined, process helps us to insure that the tool is deemed to be fair by both faculty and residents.

9. Link the overall curriculum objectives to the assessment model. We use the results of our assessments to improve, modify, and revise our curriculum. Areas of strength or weakness in teaching or assessing can provide valuable formative feedback to the program.

10. Align the goals and objectives of the Task Force with the Strategic goals for the Department, the Hospital, and the College of Medicine. Many of the strategic initiatives of the Department, Hospital, and College of Medicine are similar to the goals of the ACGME competency project. For example, patient safety parameters can be improved through better systems based and practice based competencies at the resident level. Improvements in communication and interpersonal skills might reduce professional liability events. More appropriate utilization of services in patient care or based upon evidence might improve cost containment. Assessments of professionalism might reduce the incidence of “problem residents” or allow formalization of a teaching structure for professionalism.