1	Williams’ Ten Rules in performance assessment Andrew G. Lee, M.D. Professor of Ophthalmology, Neurology, and Neurosurgery The H. Stanley Thompson Neuro-ophthalmology Clinic The University of Iowa Hospitals and Clinics
2	Williams’ Ten Commandments The 10 commandments of performance appraisal provide a foundation for the surgical training process R.G. Williams, PhD Department of Surgery Southern Illinois University School of Medicine
3	Williams’ Ten Time: valuable & limited Test knowledge application not facts Beware situation & case specificity Define blueprint & align with testing Multiple tools, multiple times, multiple observers 6. Teach & Assess at same time 7. Test score meaning > # 8. Give credit when due 9. Give feedback (formative & summative) 10. Share information with learner
4	Assessment time is a limited and precious commodity Don't test unless prepared to act on results Sample systematically & broadly Use most time-efficient assessment (but reliable & valid) method Assess only important competencies
5	Don’t bite off more than you can chew!
6	2. Test application of knowledge & skills rather than simple facts Test ability to use knowledge Test “knowledge only” as proxy to determine if fund of knowledge is limiting step
7	3. Human performance is highly situation & case specific Test for tasks & situations systematically Good performance in one situation ≠ good in all situations (halo effect) Even if competencies seem closely related Pilots licensed to fly 727s or 747s, not given unlimited pilots' licenses
8	4. Define assessments clearly, explicitly, ahead of time Use clearly articulated blueprint & behavioral objectives as guideline for test development Make sure tests accurately mirror blueprint Align testing activities with objectives requires commitment & discipline
9	5. Use variety of assessment methods (multiple) No 1 assessment measures all objectives (multiple assessments) No 1 score predicts performance (multiple tools) Pick right assessment tool for job (validity) Avoid tendency to test qualities that are easily measured Different appraisers focus on different aspects of performance (multiple observers = 360 degree)
10	6. Make every assessment a learning experience Give maximum feedback for each assessment Teach & Assess simultaneously Use rich multi-dimensional profile of performance, as opposed to a single grade (summative) Use assessment events to achieve instructional ends Use an imprecise (over no) measure of a competence if the competence being measured is highly important
11	7. Test meaning > # value Look beyond what they do & learn why they do it Behavior interpreted/understood (give meaning) Compare scores of other examinees (cohort, peer benchmarked, norm referenced) Pre-define levels of performance (criterion based, behavior anchors, scoring rubric) Improving score on same task over time Persons with similar scores have fared in practice (“as good as Resident x”)
12	8. Recognize & give credit for unanticipated learning outcomes Overly structured assessment plans can result in failure to recognize creativity & unusual achievements of learners (i.e., be flexible) Give credit for what has been learned Recognize achievement of learning outcomes
13	9. Provide high quality feedback Give feedback that is situation specific, descriptive, & constructive (formative vs. summative feedback) What did the learner do? In what context? Why was their performance desirable or undesirable? How could the situation be handled more effectively (improved)? Document improvement over time
14	10. Share all statements of appraisal with the learner Sharing => appraisals: fair, descriptive, & constructive Promotes self reflection & continuous learning Makes learner main beneficiary of your appraisal Portfolio documentation
15	Williams Ten Commandments Time: valuable & limited Test knowledge application not facts Beware situation & case specificity (halo effect) Define blueprint & align with testing Multiple tools, multiple times, multiple observers 6. Teach & Assess at same time 7. Test score meaning > # 8. Give credit when due 9. Give feedback (formative & summative) 10. Share information with learner
16	Thanks for your attention