1	UIHC GME update on the ACGME General Competencies Andrew G. Lee, MD Professor of Ophthalmology, Neurology, and Neurosurgery University of Iowa Hospitals and Clinics
2	Vision vs. Action ACGME Vision disconnected from Action = Daydream Local Action disconnected from “Vision” = Nightmare
3	Five objectives Brief recap of status of ACGME outcome project (aka: the ACGME competencies) Introduce practical general strategies Provide insights from our local UIHC efforts Two take home tools for two “tough ones” (ie, Practice-based & Systems based learning) Tell you about our UIHC local survey of status quo
4	This talk will help you even if you are….. 1. Neophyte to competencies: Introduction 2. Newly converted: Tips for starting 3. Early adopter: Tools to take home 4. True believer: Implementation matrix 5. Non-believer: Reason to believe
5	Objective 1: ACGME competencies review & update ACGME competencies outcomes project Define ACGME general competencies
6	Who is ACGME?....Depending on your perspective
7	The recap….Six general competencies Patient Care Medical Knowledge Professionalism Systems-based Practice Practice-based Learning & Improvement Interpersonal & Communication skills
8	Objective 2: General strategies Tool development (i.e., good, fast, cheap)
9	Ideal tool: Educational jargon Good Reliable (reproducible, accurate) Valid (measures what you want to measure) Feasible Generalizable Fast Low faculty burden Easy to grade and collect data Cheap Cheap
10	Improving an assessment tool: Global evaluation (A true story) Advantages Familiar User friendly Easy to use Cheap Semi-quantitative Universal (the “hammer” of evaluation)
11	Tempting to use global evaluation to assess all 6 competencies (hammer)
12	Old global evaluation form (“nine page RRC form”) Nine page form “bit off more than it could chew” (i.e., “too many questions”) Tried to measure all 6 competencies in one form Not specific for behaviors being tested Not valid (can not measure some competencies) Not reliable Not feasible Faculty revolt
13	Improving an assessment tool: Global evaluation (A true story) Disadvantages Grade inflation Halo and pitchfork effects Narrow range of scores Not able to measure some competencies Summative not formative feedback
14	Halo & pitchfork effect (“good guy” or “bad boy” syndrome)
15	Why we needed to change content of the global evaluation tool Grade inflation degrades meaning of scores Everyone is “above average” Not useful for formative feedback Resident self –esteem damaged by “low” or even average scores Faculty inter-rater reliability problems Easy and hard graders Lazy graders (“7” all the way down the form)
16	Actual faculty feedback on “nine page official RRC evaluation form” “This form takes too long to complete” Everyone gets a “7”or an “8” Half the questions end up “not applicable” “How can we possibly know if the resident is competent in practice based learning?” “What is practice based learning?”
17	Old model: Driving example Norm (1-5) referenced (peer benchmarked) Below average Average Above average Excellent Outstanding 50% of the drivers in this room are below average…raise your hand if you are “below average”!
18	Changing the scale (Criterion referenced) Dreyfus model of achieving expertise Novice Beginner Advanced beginner Proficient Expert
19	Changing the scale: Driving example Dreyfus model of achieving expertise Novice Driver’s education Beginner First year driver Advanced beginner First 5 years Proficient Next 5 years Expert > 10 years
20	Global evaluation form modifications: Summary Reduced “nine page form” to one page Reduced faculty burden and time Reduced faculty complaints Increased intra-rater reliability & inter-rater reliability Provided summative feedback
21	How we changed the tool Old paradigm Norm referenced (peer benchmarked) Summative encounter RRC “Nine-page” global evaluation form (“N/A”) Faculty burden high Creates incentives for grade inflation New paradigm Criterion referenced Formative feedback (allows for improvement) One page Faculty burden low
22	Practical lessons learned Less is more (ask less questions from more observers improves reliability) Formative is superior to summative feedback Validity is improved by measuring specific competencies assessed by behaviors Dreyfus model allows for behavior specific achievement over time
23	Where are you and your program on Dreyfus scale for the Competencies? Novice Knows about competencies Beginner Implemented rudimentary assessment tool (global evaluation form) Advanced Implementation matrix Proficient Pilot testing Expert Scholarly productivity (presentations, papers)
24	Objective 3: Insights from UIHC experience Over-arching principles Pick your allies Be a champion Pick your battles Less is more Start slow Build on success Teach & assess simultaneously Shift to learner driven documentation
25	Pick your allies Sing to the choir first You know who you can count on in your department Start a small Task Force on the competencies Meet in person (regularly) Develop written goals & objectives Implement Divide (the labor) and conquer (small steps)
26	Be a champion You have to believe (or they wont) You have to organize the effort Tools will not build themselves No one will hand them to you either Roles Lead Task Force Delegate tasks (small) Cheerleader
27	Pick your battles You know who will do what and when e.g., Ask senior people to teach but not assess, ask volunteer faculty to assess but not teach Small bites Don’t ask everyone to do everything Don’t ask a few to do everything Be realistic
28	Less is more Ask fewer questions of more individuals Time burden limit: 15 minutes e.g., Global Evaluation form = one page Minimize number of competencies assessed at one encounter Validate tools against one another over time Discriminative validity (PGY 2 vs. PGY 4) Concurrent validity (different tools same time)
29	Start small Pick low hanging fruit first Start with validated tools Start with easy tools Written (in-service) exams Global evaluation form Journal club (practice based learning) Portfolio
30	Develop an implementation matrix Explicit objectives Roadmap of achievement Inventory of existing educational tools Checklist and timeline Follow your implementation matrix
31	Implementation matrix: Best practices & “Blue sky” Multiple assessments Multiple observers Multiple tools Formative & summative purposes Adequate timing & frequency of assessment Good (reliable, valid), fast, & cheap!
32	Teach & assess simultaneously Use existing educational inventory Teach & assess at the same encounter Grand rounds = practice based learning Staffing clinic = patient care Informed consent = professionalism, communication skills Morbidity conference = systems based care Chart rounds = practice based learning
33	Matrix contents: Blue print Why are you assessing? (importance?) Who will assess the learner? When will the assessment occur? What tool(s) will be used? How will the assessment be used (formative, summative, both)? How often will the assessment occur?
34	Objective 4: Two take home tools for Two difficult competencies Two take home tools Structured journal club Portfolio Two tough competencies Practice based learning Systems base learning
35	Define PBL competency Practice-Based Learning and Improvement Investigation & evaluation of own patient care Appraisal & assimilation of evidence Improvements in own patient care (Evidence based medicine in practice)
36	Example: Journal club Why: To assess practice based learning Who: Rotating faculty journal club leader When: Four times per year What: Structured journal club checklist How: Formative & summative encounter
37	Results of pilot study N=29 residents 3 academic centers (Iowa, UC, UCLA)
38	Scoring rubric: (Pre and post self-testing) 1 = Unaware of concept in PBL 2 = Aware but did not apply 3 = Applied did not document results 4 = Documented application but did not sustain 5 = Sustained application of concept over time
39	Five domains A: Appraise and assimilate scientific evidence B: Read critically & draw clinical conclusions C: Use systematic & standardized checklist D: Apply study design & statistical methods E: Maintain self-documented written record (e.g., learner portfolio) of PBL
40	Percentage increase in scores for five domains (A through E): n = 29 Increase in score between pre & post testing Domain P value A 51% 0.0005 B 72% < 0.0001 C 27% < 0.0001 D 72% < 0.0001 E 86% < 0.0001
41	Score change by domain
42
43	Bottom line: Journal club Statistically significant improvement (p< .0001) Resident self-assessed PBL scores improved across all 5 ACGME domains
44	Conclusions: Journal Club Journal club can be used as a tool to both TEACH & ASSESS competency JC is well known, familiar, off the shelf, feasible, valid, reliable, & cheap JC in our pilot study showed statistically significant improvement in self-assessed competence
45	Take home message The journal club tool is… Fast (off the shelf, in your program now) Cheap (free) Good (reliable, valid, feasible)
46	Defining SBL competencies Systems-Based learning (SBL) Awareness of and responsiveness to larger context & system of health care Ability to effectively call on system resources to provide care that optimal (Work within the health care system)
47	What is a portfolio? Everything a learner needs to accomplish Road map for success Everything a learner does Chronicles milestones Highlights “best work” Permanent record Not just a repository
48	Why Portfolio? Assessment drives learning Competencies require documentation Reliable, valid, feasible, & inexpensive tools needed for assessment Need for robust “off the shelf” tools now Focus on individual quality improvement
49	Why portfolio is a great tool? Great “measuring tape” for multiple competencies Can measure “tough” ones (e.g., practice-based & system based learning) Draws from actual work in real world Inexpensive to implement Learner driven
50	Basic skeleton for portfolio What’s in it…. Section 1 - General Demographics Section 2 - Credentialing Documentation Section 3 - Department Credentials Section 4 - Examination Scores Section 5 - Publications / Presentations Section 6 – Evaluations Section 7- Self assessment project (follow patient through system of health care, track costs of care) Section8 – PBL chart audit/chart review
51	When and How can I implement? Start now Explicit criteria for contents Write down what you want Start small (skeleton first) Modify as needed (add more meat to skeleton)
52	The future… Take home CD-ROM On line portfolios PDA input (multimedia) Direct real time access of faculty & program director to portfolio data Comparison of aggregate data Testing of reliability & validity
53
54	Teach AND Assess simultaneously (Newcastle coals?) Existing tools Journal club Morbidity conference Global evaluation Grand rounds Summative feedback Surgery/procedure logs Revised tools Structured journal club Near Miss analysis 360 degree evaluation Record presentations include references Formative feedback Portfolio
55	New tools for the toolbox Case based learning Direct observation: simulated, standardized or real patients (objective structured clinical exam: OSCE) Journal club M & M (Near miss) Chart audit or chart stimulated recall Qualitative reviews: Patients, ancillary, peers (not just faculty) Portfolio (surgical & procedure logs)
56	Matrix matching tools to competencies Patient care Medical knowledge Practice based learning Interpersonal skills Professionalism Systems-based practice Surgery Direct observation, patient surveys Written & oral exams Journal club, quality assurance chart audit Observation, evaluations Performance reviews Web based vignettes, case studies, portfolio, near miss Surgical OSCE, video or virtual surgery, portfolio
57	What is the matrix?
58	ACGME: What is the matrix?…what will your answer be??….
59	Be Bold, Be Fearless…
60	Blending in to the crowd wont work
61	Objective 5 Survey results forthcoming Where are you in the outcomes project (novice, proficient, expert?) How can GME help you? What do you want? What do you need?
62	What can GME offer you? Clearinghouse of best practices Compare and contrast existing tools Compliance consultation(s) for programs Compile pooled educational resources Collect shared tools & experience Communicate via web resources Chat room Call the help line (ask the experts) Coordinate multi-departmental pilot testing
63	Summary Recap of status of ACGME outcomes project (aka: the competencies) Practical general strategies & insights from our local UIHC efforts Few specific approaches for Practice-based learning and Systems based learning Give you two take home tools (JC, Portfolio) Tell you about our local survey of status quo
64	Hopefully….Vision AND Action 1. Neophyte to competencies: Introduction 2. Newly converted: Tips for starting 3. Early adopter: Tools to take home 4. True believer: Implementation matrix 5. Non-believer: Reason to believe