SCIENCE@home

Resident competence assessment: Best practices

K. Golnik

Published 2016 in Journal of Current Ophthalmology

ABSTRACT

Competence can be defined as “the ability to do something well.” The goal of ophthalmology residency training is to produce competent ophthalmologists. To prove competence, appropriate assessment methods are required. These methods should follow certain principles of assessment.1 Teaching ophthalmologist's time is valuable and limited and often poorly or uncompensated. Thus any assessment should be as time-efficient as possible but still provide valid and reliable information. However, the easiest and fastest assessment methods (written tests) are not appropriate for assessment of skill. Additionally, performance is situation specific. Good performance on medical knowledge multiple choice test does not mean there will also be good performance in procedural skills or examination techniques. Thus situation specific assessments must be used. Assessment methods should be known by the residents and aligned with objectives and performance expectations. Assessments can also be a learning experience. This may not always be possible if the assessment is an examination that leads to passing or failing a course, rotation or residency. This type of assessment is known as summative feedback. On the contrary, formative feedback is assessment that is designed to improve performance and thus must be shared with the resident. The vast majority of assessments should be formative. In recent years a variety of ophthalmology-specific competency assessment tools have been developed. The International Council of Ophthalmology (ICO) offers the only internationally developed test of ophthalmic medical knowledge. The ICO examinations include both basic and advanced levels.2 The Ophthalmic Clinical Evaluation Exercise (OCEX) is designed to assess resident competence in evaluating a new patient. The teaching physician completes scoring in 33 categories that rate the residents ability to communicate effectively, perform a history and examination, and synthesize the information into a differential diagnosis and plan. The OCEX has been shown to have content validity and inter-rater reliability.3, 4 Several methods of surgical skill assessment have been devised. Cremers and associates developed the “Objective Assessment of Skills in Intraocular Surgery” (OASIS), a one-page objective evaluation form to assess residents' skills in cataract surgery.5 The form is completed by an evaluator who directly observes the surgical procedure and includes objective data such as wound placement and size, phacoemulsification time, and total surgical time, etc. To complement this objective assessment the same group developed a subjective rating of surgical skills named “Global Rating Assessment of Skills in Intraocular Surgery” (GRASIS).6 This one-page form allows the evaluator to assign scores from 1 to 5 based on a behaviorally anchored rubric to domains such as pre-operative knowledge, microscope use, instrument handling, and tissue treatment in addition to seven other areas. Saleh and colleagues described an assessment tool called the “Objective Structured Assessment of Cataract Surgical Skill” (OSACSS).7 This tool breaks down the phacoemulsification procedure into 20 steps that are scored on a 5-point Likert scale. The ICO modified this tool to include a modified Dreyfus model of skill acquisition (novice, beginner, advanced beginner, competent) and description of the skill required to achieve each level in each step of the surgical procedure.8 Once drafted, content and face validity were achieved by having an international panel of 15 experts review the draft instrument and provide feedback. The product is the internationally valid and reliable ICO-Ophthalmology Surgical Competency Assessment Rubric (OSCAR) – phacoemulsification.8, 9 In a similar fashion internationally applicable assessment tools for extracapsular cataract surgery (ICO-OSCAR:ECCE),8 small incision cataract surgery (ICO-OSCAR:SICS),10 lateral tarsal strip surgery (ICO-OSCAR:LTS),11 and strabismus surgery (ICO-OSCAR:strabismus)12 were developed. Additional ICO-OSCARS are being developed for panretinal photocoagulation, corneal transplant, vitrectomy, trabeculectomy and pediatric cataract surgery. Medical educators have also been devising methods to teach and assess the competencies other than patient care and medical knowledge. Lee and associates described an assessment tool involving a structured assessment of journal club that leads to assessment of practice based learning and improvement.13, 14 Golnik and associates developed and showed validity of a tool to assess resident on-call performance, the “On Call Assessment Tool” (OCAT).15 The OCAT is a one page checklist to be used retrospectively during random chart review of on-call consultations. Finally, a “360°” evaluation tool can be very helpful in evaluating professionalism and communication skills. The intent is to get feedback on residents from all groups with which they interact. Thus, faculty evaluations can be considered 90°, patient surveys, ancillary staff and peer evaluations constitute the other 270°. Self-evaluations may be utilized in addition. Of course, patients and staff are not asked about the resident's medical knowledge but rather about professionalism and communication skills. Jagadeesan and associates have shown their patient satisfaction survey can discriminate levels of resident communication skill and thus may be useful to assess this competency.16 No internationally valid 360-degree assessment tool exists and thus the ICO is currently developing one in a manner similar to the ICO-OSCARs. In this issue Hassanpour and associates compared video observation of procedural skills (VOPS) to direct observation of procedural skills (DOPS) using a 10-point Likert scale. They also investigated intra-rater reliability. They found good correlation between VOPS and DOPS scores and very good intra-rater reliability. The VOPS method has the advantage of completion at a later time and may be more convenient. It also could lead to multiple assessors, and studies of inter-rater reliability should be considered. It would be interesting to apply the soon to be published ICO-OSCAR for trabeculectomy in this paradigm. Both the VOPS and DOPS methods meet the principles of good assessment described above. Utilization of tools such as these are essential in producing and assuring competence of our future ophthalmologists.

PUBLICATION RECORD

CITATION MAP

EXTRACTION MAP

CLAIMS

  • No claims are published for this paper.

CONCEPTS

  • No concepts are published for this paper.

REFERENCES

Showing 1-14 of 14 references · Page 1 of 1

CITED BY

Showing 1-6 of 6 citing papers · Page 1 of 1