The Reliability of Computer Software to Score Essays: Innovations in a Humanities Course

In the summer of 2001, Florida Gulf Coast University was awarded a 2-year, $200,000 grant from the National Center for Academic Transformation to redesign a required General Education course entitled Understanding the Visual and Performing Arts. The course redesign project had two main goals: infuse appropriate technology into the course in meaningful ways and reduce the cost of delivering the course. Faculty members in the humanities and arts were adamant that the redesigned course be structured in such a way that it offered a coherent and consistent learning experience for all students and that it maintained the use of essays as an important strategy for learning in the class. The redesign project led to the creation of a wholly online course with all students registered in two large sections. One of the ways in which we continued to incorporate essay writing into the course was to use a computer application, the Intelligent Essay Assessor (IEA) from Pearson Knowledge Technologies, to score two shorter essays. Through detailed assessment, we have demonstrated that the computer software has an inter-rater reliability of 81% as compared to the 54% inter-rater reliability of the holistic scoring by humans. In this essay, we provide general background on the redesign project and a more detailed discussion of the appropriate use and the reliability of the Intelligent Essay Assessor.