Exploring Rater Effect with Item Response Models

• Main Authors: Cheng-Te Chen, 陳承德
• Other Authors: Shu-Ying Chen
• Format: Others
• Language: en_US
• Published: 2009
• Online Access: http://ndltd.ncl.edu.tw/handle/90128196964491540010

博士 === 國立中正大學 === 心理學所 === 97 === The judgment from raters suffers from many biases; hence this dissertation extends current item response models to incorporate the inter-rater variability, intra-rater variability, differential rater functioning, and multidimensional framework for explaining these rater biases. This study first reviews several rater biases (effects) that exhibited in the rater data. Second, a detailed discussion on the extended models or solutions is provided. Next, the focus of this study is centered on four main topics: (a) the comparison of two recently developed models (RE-MFM and HRM) that both explain the variation between and within raters but had not been fully compared are made here; (b) the nature of random sample for rater and ratee is included into the current rater models, so the generalization of test result is extended; (c) the models accounting for the different rater functioning (DRF) are proposed and methods to detect DRF raters are examined; (d) the multidimensional extensions of rater models are developed, and the parameter recovery are assessed. The feasibility of models is evaluated through simulation studies. However, the proposed models contain several random effects and dimensions. Fortunately, current software WinBUGS is available for these models; no efforts are needed to develop parameter estimation procedures. The simulation results showed that (a) the RE-MFM is more flexible and comprehensive than the HRM; (b) the nature of random sample can be successfully incorporated into the rater models and the distributional parameters of rater and ratee pool can be well recovered; (c) the suggesting method for identifying DRF rater has a high detecting power with a well-controlled Type I error rate; (d) the parameters of multidimensional extensions of rater models are well-recovered. In the following, an application of each proposed rater model is implemented through an empirical example. The results of this study are summarized and discussions are brought in the last part. This study opens the possibility of integrating the more and more complex rater problems into the item response framework. Further explorations and examinations are demanded to solve, to explain, to prevent, and to remedy all kinds of rater problems for an accurate, meaningful, and less biased score.