On the relation between automated essay scoring and modern views of the writing construct

This paper examines the construct measured by automated essay scoring (AES) systems. AES systems measure features of the text structure, linguistic structure, and conventional print form of essays; as such, the systems primarily measure text production skills. In the current state-of-the-art, AES provide little direct evidence about such matters as strength of argumentation or rhetorical effectiveness. However, since there is a relationship between ease of text production and ability to mobilize cognitive resources to address rhetorical and conceptual problems, AES systems have strong correlations with overall performance and can effectively distinguish students in a position to apply a broader writing construct from those for whom text production constitutes a significant barrier to achievement. The papers begins by defining writing as a construct and then turns to the e-rater scoring engine as an example of AES state-of-the-art construct measurement. Common criticisms of AES are defined and explicated—fundamental objections to the construct measured, methods used to measure the construct, and technical inadequacies—and a direction for future research is identified through a socio-cognitive approach to AES.

► Examines the construct measured by automated essay scoring (AES) systems.
► Establishes and explicates common criticisms of AES.
► Establishes limits of AES in assessing argumentation and rhetorical effectiveness.
► Explains relationship between text production and cognitive resources.
► Established directions for future research through a socio-cognitive approach to AES.