Data with hierarchical structure: impact of intraclass correlation and sample size on Type-I error

• Main Authors: Serban C Musca, Rodolphe eKamiejski, Armelle eNugier, Alain eMéot, Abdelatif eEr-rafiy, Markus eBrauer
• Format: Article
• Language: English
• Published: Frontiers Media S.A. 2011-04-01
• Series: Frontiers in Psychology
• Subjects: correction for nonindependence of observations; groups nested under treatment; hierarchical data structure; multilevel modeling; Type-I error
• Online Access: http://journal.frontiersin.org/Journal/10.3389/fpsyg.2011.00074/full

Least squares analyses (e.g., ANOVAs, linear regressions) of hierarchical data leads to Type-I error rates that depart severely from the nominal Type-I error rate assumed. Thus, when least squares methods are used to analyze hierarchical data coming from designs in which some groups are assigned to the treatment condition, and others to the control condition (i.e., the widely used "groups nested under treatment" experimental design), the Type-I error rate is seriously inflated, leading too often to the incorrect rejection of the null hypothesis (i.e., the incorrect conclusion of an effect of the treatment). To highlight the severity of the problem, we present simulations showing how the Type-I error rate is affected under different conditions of intraclass correlation and sample size. For all simulations the Type-I error rate after application of the popular Kish (1965) correction is also considered, and the limitations of this correction technique discussed. We conclude with suggestions on how one should collect and analyze data bearing a hierarchical structure.