Just Ask Teachers: Building expertise, trusting subjectivity, and valuing difference in writing assessment

• Calibrated scoring overemphasizes reliability.
• Developing teacher expertise and program improvement requires trusting teacher subjectivity.
• Expertise involves intuition and subjectivity.
• Writing assessment should involve numerous messy factors rather than be reduced and simplified to a rubric.

The authors theorize a method for writing assessment that deemphasizes the traditional privileging of validity and reliability generated from multiple-reader, calibrated scoring of samples of student work. While acknowledging the holistic model's benefits to the field of writing studies, the authors assert that its claims of accuracy and objectivity minimize the numerous tangible and intangible variables that writing teacher/experts understand and value as they evaluate writing. The removal of the “object” – writing artifact – from its context in order to assess it quantitatively diminishes the opportunities for achieving meaningful and pedagogically effective results for a writing program. Rather than calibrating teachers to a rubric, the proposed method here generates a rough calibration of teacher “values” via facilitated conversations, accepting the differences of opinions and “messiness” of teachers’ subjective views of writing. Teachers then periodically assess their students’ performance on these values as well as the course objectives. In this way, the process develops teacher contextual expertise while producing focused assessment data that is both useful for outside agencies and meaningful to the program's goals of improving the teaching of writing.