| Summary: | Evidence based on content is one of the key sources of validity evidence described in the Standards for Educational and Psychological Testing. This dissertation used three groups of content experts to gather validity evidence based on content for two sections of the Quality of Life for Homeless and Hard-to-House Individuals (QoLHHI) Survey. One goal of this dissertation was to examine the implications of this evidence for the validity of inferences made from the QoLHHI, while a second goal was to determine if judgmental studies using content experts, a popular method for content validation, can be improved.
Quantitative and descriptive validity evidence was collected from 11 subject matter experts (researchers working with individuals who are homeless and vulnerably housed (HVH)), 16 experiential experts (individuals who were HVH), and 8 practical experts (individuals who had administered the QoLHHI). These experts independently rated the relevance and technical quality of the items, response scales, administration instructions and other content elements of the QoLHHI. Content Validity Indices were computed and used to identify elements that were endorsed by the experts. Experts also provided descriptive feedback in the form of comments and suggestions for improving the QoLHHI content. This feedback was used to develop recommendations for revising the content.
In terms of evidence based on content for the QoLHHI, over 85% of the content elements were endorsed, indicating good evidence for validity. The recommended revisions are relatively straightforward to implement. Overall, the content of these two sections of the QoLHHI appears likely to produce scores that lead to supportable and appropriate inferences.
With regards to methodology, a new approach to content validation is proposed. Traditionally, content validation studies have focused on quantitative assessments, with descriptive data being given a secondary role. It is proposed here that content experts should be viewed as an advisory board rather than a representative sample of a larger population, that members’ feedback should be given individual attention, and that judgmental studies should focus primarily on the experts’ descriptive feedback, which is where their knowledge, experience, and insight are best expressed.
|
|---|
