Psychol Methods. 2026 Jul 20. doi: 10.1037/met0000860. Online ahead of print.
ABSTRACT
The ability of a student can be conceptualized as either a continuously varying entity (e.g., conventional analysis using dichotomous item response theory [IRT] models; Lord & Novick, 1968) or a bundle of latent classes (e.g., cognitive diagnostic models [CDMs]; Rupp et al., 2010; von Davier & Lee, 2019). This article builds on recent efforts to focus on predictive differences between measurement models in an attempt to examine the degree to which such approaches, which utilize quite distinctive notions regarding the nature of ability, produce different predictions of response behavior in the real world. We first present two simulation studies in which data are generated from variants of CDMs that differ in sample size, attribute hierarchical structures, and attribute estimation methods. We illustrate that, as we would expect given that they were used to generate data, CDM-based predictions uniformly outperform those of IRT models. We then compare the performance of CDM- and IRT-based approaches across 11 empirical data sets previously analyzed using CDMs. Our findings indicate that overfitting is a pervasive issue across CDM-based predictions, particularly with the generalized deterministic inputs, noisy “and” gate model. Furthermore, in no case does the CDM show superior performance when using the maximum a posteriori estimator, and only six out of 11 data sets show improved model fit for CDMs over the two-parameter logistic model when using the marginal mastery probabilities estimator. Researchers and practitioners may need to balance the diagnostic appeal of CDMs with the fact that their complexity can come at the cost of predictive accuracy. (PsycInfo Database Record (c) 2026 APA, all rights reserved).
PMID:42475042 | DOI:10.1037/met0000860
