Easily too difficult: Estimating item difficulty in computer simulated microworlds

Dealing with complexity and dynamics is increasingly becoming part of people's everyday lives. The necessity of dealing with complex systems has instigated the use of computer simulations, so-called microworlds (MWs), to assess and study human behavior in complex situations. Although these MWs enjoy great popularity with both practitioners and researchers, their psychometric qualities have been questioned, and studies that have investigated these qualities have been sparse. In particular, only a few studies have investigated the factors that contribute to item difficulty in MWs. To fill this gap, we analyzed data from 3128 Finnish students with a linear logistic test model. Our results suggest that item difficulty in MWs can be almost perfectly predicted by six basic item characteristics, namely, (a) the use and number of eigendynamics, the numbers of (b) input and (c) output variables, the numbers of (d) input and (e) output variables not related to any other variables, and (f) the total number of relations between all variables. In addition, we provide evidence for the necessity of differentiating between the difficulty of controlling an MW (knowledge application) and understanding its underlying structure (knowledge acquisition). Finally, we discuss further theoretical and practical implications of an increased understanding of MWs for their use as assessment instruments.