Psychometric analysis of the performance data of simulation-based assessment: A systematic review and a Bayesian network example

• We present a review on psychometric analysis of simulation-based assessment.
• We performed the review from the Evidence-Centered Design framework perspective.
• The Bayes Net is the most used psychometric model for simulation-based assessment.
• We present an example of a Bayes Net of a real simulation-based assessment.
• We make recommendations regarding the development of a simulation-based assessment.

Researchers have shown in multiple studies that simulations and games can be effective and powerful tools for learning and instruction (cf. Mitchell & Savill-Smith, 2004; Kirriemuir & McFarlane, 2004). Most of these studies deploy a traditional pretest-posttest design in which students usually do a paper-based test (pretest) then play the simulation or game and subsequently do a second paper-based test (posttest). Pretest-posttest designs treat the game as a black box in which something occurs that influences subsequent performance on the posttest (Buckley, Gobert, Horwitz, & O'Dwyer, 2010). Less research has been done in which game play product data or process data itself are used as indicators of student proficiency in some area. However, the last decade researchers have started focusing on what is happening inside the black box to an increasing extent and the literature on the topic is growing. To our knowledge, no systematic reviews have been published that investigate the psychometric analysis of performance data of simulation-based assessment (SBA) and game-based assessment (GBA). Therefore, in Part I of this article, a systematic review on the psychometric analysis of the performance data of SBA is presented. The main question addressed in this review is: ‘What psychometric strategies or models for treating and analyzing performance data from simulations and games are documented in scientific literature?’. Then, in Part II of this article, the findings of our review are further illustrated by presenting an empirical example of the – according to our review – most applied psychometric model for the analysis of the performance data of SBA, which is the Bayesian network. Both the results from Part I and Part II assist future research into the use of simulations and games as assessment instruments.