Student perception and usage of an automated programming assessment tool

Automated assessment systems are gaining popularity within computer programming courses. In this paper we perform an empirical evaluation of Mooshak, an online judge that verifies program correctness, in order to determine its usefulness in classroom settings. In particular, we provide a detailed study on how students use the tool, analyze their opinions and critiques about it, and measure other features like its capability to reduce dropout rates. The experience was carried out within a course on algorithm design and analysis where we collected information through several questionnaires and data generated by the tool during the course. Among the main findings we highlight: (1) the usage of the tool was adequate in relation to students' own testing; (2) its feedback needs to be richer in order to improve its acceptance among students; and (3) there was no statistical evidence to claim Mooshak reduced the dropout rate.