On the vulnerability of automated scoring to construct-irrelevant response strategies (CIRS): An illustration

• Illustrated vulnerability of automated scoring to construct-irrelevant response strategy.
• A strategy using longer, less frequent words improved scores of low-scoring essays.
• The strategy had opposite effect on high-scoring essays.

This research is motivated by the expectation that automated scoring will play an increasingly important role in high stakes educational testing. Therefore, approaches to safeguard the validity of score interpretation under automated scoring should be investigated. This investigation illustrates one approach to study the vulnerability of a scoring engine to construct-irrelevant response strategies (CIRS) based on the substitution of more sophisticated words. That approach is illustrated and evaluated by simulating the effect of a specific strategy with real essays. The results suggest that the strategy had modest effects, although it was effective in improving the scores of a fraction of the lower-scoring essays. The broader implications of the results for quality assurance and control of automated scoring engines are discussed.