Does Item Exposure Impact the Relationship Between Specificity and Interrater Agreement

• Main Author: Littlejohn, Traxler
• Format: Others
• Published: TopSCHOLAR® 2006
• Subjects: Psychology
• Online Access: http://digitalcommons.wku.edu/theses/292; http://digitalcommons.wku.edu/cgi/viewcontent.cgi?article=1295&context=theses

The present study attempted to replicate the results found in Roch and Paquin (2004). Raters in their study viewed a video tape performance and rated a target's performance during a leaderless group discussion. Their results indicated that as item specificity decreased (i.e., as items became more vague), there was an increase in interrater agreement, a positive correlation between item difficulty and interater agreement, and a positive correlation between performance and interrater agreement. The present study also examined the impact of item exposure as it related to these hypotheses and the relationship between item exposure and interrater agreement. Using a sample of 299 participants, the present study used the same video of a leaderless group discussion, rating form, and specificity ratings as used in Roch and Paquin (2004). The results of this study found a positive correlation between specificity and agreement and that item exposure significantly positively impacted interrater agreement.