An examination of several in-basket scoring strategies and their effect on reliability and criterion-related validity

• Main Author: Harlos, Karen P.
• Language: English
• Published: 2008
• Online Access: http://hdl.handle.net/2429/1373

The in-basket exercise, a paper-and-pencil measure of administrative ability, is an assessment technique characterized by complex, often subjective scoring procedures which have limited wide-scale applications of this popular instrument. Because the literature affords no clear classification system for this and related exercises, a structure which clarified the definitions of and relationships among management games, management simulations, work sample tests, and in-basket exercises was introduced. The primary purpose of the present research was to investigate the cross-sample generalizability of several strategies for scoring the in-basket exercise, including a reduced-item scoring approach in which an optimal subset of items was identified for scoring. The preliminary studies upon which the present research is based are also discussed in the present work. In addition to examining the impact of these scoring strategies on reliability and criterion-related validity, consideration was given to addressing long-standing concerns of in-basket training- and scoring time demands. Three hundred and twenty-one entry-level employees from a large western Canadian utility company were administered the same in-basket exercise previously applied in a different Canadian utility company. Contrary to expectations, the shrinkage in validity using an empirically-based scoring key was substantial, pointing to the selection of a more logically-derived panel key as the method of choice. The introduction of a new cognitive-based measure of in-basket performance showed promising results. In addition, the reduced-item scoring approach did not result insignificant losses in reliability or criterion-related validity, thus allowing substantial reductions in training- and scoring-time. Implications for in-basket development are also considered.