How games for computing education are evaluated? A systematic literature review

Educational games are assumed to be an effective and efficient instructional strategy for computing education. However, it is essential to systematically evaluate such games in order to obtain sound evidence of their impact. Thus, the objective of this article is to present the state of the art on how games for computing education are evaluated. Therefore, we performed a systematic literature review of a sample of 3617 articles from which 112 relevant articles have been identified, describing 117 studies on the evaluation of games for computing education. Based on these studies we analyzed how evaluations are defined (the analysis factors evaluated, research designs, evaluation models/methods used, kind of data collection instruments, etc.), how they have been executed (sample size and replications) and analyzed (data analysis methods used). As a result, we can confirm that most evaluations use a simple research design in which, typically, the game is used and afterwards subjective feedback is collected via questionnaires from the learners. The majority of the evaluations are run with small samples, without replication, using mostly qualitative methods for data analysis. We also observed that most studies do not use a well-defined evaluation model or method. This shows that there is a need for more rigorous evaluations as well as methodological support in order to assist game creators and instructors to improve such games as well as to systematically support decisions on when or how to include them within instructional units.