Does writing development equal writing quality? A computational investigation of syntactic complexity in L2 learners

• Significant growth in syntactic complexity occurs in L2 writers as a function of time spent studying English.
• L2 writers produced syntactic structures that are increasingly aligned with academic writing over time.
• Human raters assess text quality based on syntactic structures aligned with spoken discourse.

This study examines second language (L2) syntactic development in conjunction with the effects such development has on human judgments of writing quality (i.e., judgments of both overall writing proficiency and more fine-grained judgments of syntactic proficiency). Essays collected from 57 L2 learners in a longitudinal study were analyzed for growth and scoring patterns using syntactic complexity indices calculated by the computational tool Coh-Metrix. The analyses demonstrate that significant growth in syntactic complexity occurred in the L2 writers as a function of time spent studying English. However, only one of the syntactic features that demonstrated growth in the L2 learners was also predictive of human judgments of L2 writing quality. Interpretation of the findings suggest that over the course of a semester, L2 writers produced texts that were increasingly aligned with academic writing (i.e., texts that contain more nouns and phrasal complexity), but that human raters assessed text quality based on structures aligned with spoken discourse (i.e., clausal complexity). Thus, this study finds that the syntactic features that develop in L2 learners may not be the same syntactic features that will assist them in receiving higher evaluations of essay quality.