Correlation of rater training and reliability in performance assessment: Experience in a school of dentistry

• Main Authors: Chien-Ju Lin, Jenny Zwei-Chieng Chang, Tan-Ching Hsu, Yi-Jen Liu, Szu-Hsien Yu, Stanley Shih-Li Tsai, Eddie Hsiang-Hua Lai, Chun-Pin Lin
• Format: Article
• Language: English
• Published: Elsevier 2013-09-01
• Series: Journal of Dental Sciences
• Subjects: checklist; performance assessment; rater training; reliability; videotape
• Online Access: http://www.sciencedirect.com/science/article/pii/S1991790213000044

Background/purpose: Well-constructed and validated evaluation tools have been used for clinical performance assessment (including objective-structured clinical examination) in many countries for years. The aim of performance assessment in dentistry is to evaluate whether dental graduates are clinically competent in essential skills, and these results can be utilized in modifying teaching and training programming. Thus, to improve the reliability of the evaluation tools, inter-rater reliability is weighted heavily. The aim of this study is to investigate the correlation between rater training and rater reliability. Materials and methods: Two sixth-year dental students who had already undergone a half-year of internship completed an 8-minute subgingival root planing procedure, and the students' performance was captured on videotape. Nine faculties from the School of Dentistry, who had participated in developing this case, were invited to observe the recorded video and to rate the two students using a checklist. One month later, after receiving further assessment training (workshop including role-play, rating practice, discussion, etc.), the same nine raters observed the same video again and re-rated the students using the same checklist. Results: Analysis results of inter-rater reliability for the two students in the initial rating were W = 0.770 and 0.763. Results of re-rating (1 month later) were W = 0.891 and 0.827. All results were statistically significant (P <0.001). Conclusion: Rater training by means of role-play, rating practice, and discussion do improve the inter-rater reliability of performance assessment in dentistry.