Producing equivalent examination forms : an assessment of the British Columbia Ministry of Education examination construction procedure

• Main Author: MacMillan, Peter D.
• Language: English
• Published: University of British Columbia 2011
• Subjects: Examinations > British Columbia; Examinations > Design and construction
• Online Access: http://hdl.handle.net/2429/31009

Questions have been raised concerning the equivalency of the January, June, and August forms of the British Columbia provincial Grade 12 examinations for a given subject. The procedure for constructing these examinations has been changed as of the 1990/91 school year. The purpose of this study was to duplicate this new procedure and assess the equivalency of the forms that resulted. An examination construction team, all of whom had previous experience with the British Columbia Ministry of Education's Student Assessment Branch, simultaneously constructed two forms of a Biology 12 examination from a common table of specifications using a pool of multiple choice items from previous examinations. A sample of students was obtained in the Okanagan, Thompson, and North Thompson areas of British Columbia. Both forms were administered to each student, as required by the test equating design (Design II (Angoff, 1971)) chosen. The data sample consisted of responses from 286 students. The data were analyzed using a classical item analysis (LERTAP, Nelson, 1974) followed by a 2x2 order-by-form fixed effects ANOVA with repeated measures on the second factor. Item analysis revealed all items on both forms performed satisfactorily, ruling out an alternate hypothesis of flawed items being the cause of the lack of equivalence found. Results showed a significant (p<.05) difference in the means of the two forms, no significant (p>.25) order effect, and a significant (p<.25) order-by-form interaction. Linear and equipercentile equatings were carried out. The linear and equipercentile equatings yielded very similar results. Equating errors, judged using the conditional root mean square error of equating, was 4.86 points (9.35%) for both the equatings. Equivalency was also judged employing a graphical procedure in which the deviation of the equating function from the identity function was plotted with error bands produced using the standard error of equating. The procedure showed the two forms to be nonequivalent, particularly for the lower scoring students. The source of the nonequivalency was investigated by separating the forms into three subtests based on the pairs of items possessing or lacking item statistics at the time of test construction. The linear equating followed by the graphical analysis was repeated for the pairs of subtests. The pairs of subtests comprised of item pairs for which difficulty (p) values were present at time of construction for one or both of the items in an item pair were found to be equivalent. In contrast, the pair of subtests comprised of items for which p values were unavailable for either item in an item pair at time of construction were found to be not equivalent. It was concluded that the examination construction procedure in its present state cannot be relied on to produce equivalent forms. An experienced examination construction team was unable to accurately match items based on the level of difficulty for items which did not have prior item statistics. As such, a necessary requirement for construction of equivalent forms is that item statistics be present at the time of construction. === Education, Faculty of === Graduate