Using psychometric technology in educational assessment: The case of a schema-based isomorphic approach to the automatic generation of quantitative reasoning items

This article deals with the investigation of the psychometric quality and constructs validity of algebra word problems generated by means of a schema-based version of the automatic min–max approach. Based on review of the research literature in algebra word problem solving and automatic item generation this new approach is introduced as a theory-based top–down method of automatic item generation featuring a quality control framework aimed to minimize the construct unrelated variance in the item parameters. The first study deals with the evaluation of an initial set of items. The results are replicated in the second study using a larger item set which also allows the investigation of the construct representation of the generated item. Since construct unrelated variance components (e.g. reading comprehension) have been controlled for in the item generation phase the results revealed some interesting insights into the cognitive processes of the actual mathematization phase of algebra word problem solving. The third study investigated the nomothetic span is using hierarchical confirmatory factor analysis. The results argue for the convergent and discriminant validity of the automatically generated items. Taken together, the results indicate that the automatic generation of construct valid algebra word problems at a high psychometric level is viable. The discussion is thus concerned with the implications of this new approach to item generation for theory development and evaluation as well as practical benefits for educational assessment and the development of intelligent tutoring systems.