Large-scale assessment, locally-developed measures, and automated scoring of essays: Fishing for red herrings?

Automated Essay Scoring (AES) has garnered a great deal of attention from the rhetoric and composition/writing studies community since the Educational Testing Service began using e-rater® and the Criterion® Online Writing Evaluation Service as products in scoring writing tests, and most of the responses have been negative. While the criticisms leveled at AES are reasonable, the more important, underlying issues relate to the aspects of the writing construct of the tests AES can rate. Because these tests underrepresent the construct as it is understood by the writing community, such tests should not be used in writing assessment, whether for admissions, placement, formative, or achievement testing. Instead of continuing the traditional, large-scale, commercial testing enterprise associated with AES, we should look to well-established, institutionally contextualized forms of assessment as models that yield fuller, richer information about the student's control of the writing construct. Such tests would be more valid, as reliable, and far fairer to the test-takers, whose stakes are often quite high.

► The limitations of Automated Essay Scoring are described, using an evaluation of the proponents’ own descriptions.
► The kinds of test AES can score are described, along with their under-representation of the writing construct and the negative consequences of their use.
► Alternatives, in the form of richer, broader assessments that yield far more information about the test taker and his or her context are suggested.
► A theoretical model for building more robust assessments is provided.