The design and analysis of small-scale syntactic judgment experiments

A growing literature argues that native-speaker syntactic judgments are best tested with standard psycholinguistic protocols, but the difficulty of formal experimentation has prevented many syntacticians from trying it. Fortunately, as shown by textual analysis of a typical theoretical syntax paper (Li, 1998), syntacticians already recognize the value of proper experimental designs and quantitative analyses, albeit in nascent forms. Small-scale judgment experiments build on this foundation by applying the minimum amount of extra effort needed to draw valid statistical inferences from the type of data most familiar to syntacticians, namely binary judgments involving very few speakers and sentences. Statistical methods appropriate for such data, some hitherto underused, are described, and software tools for automating small-scale judgment experimentation are introduced. The simplicity and power of the methods are then illustrated in a small-scale test of the claims in Li (1998) on naive Chinese speakers.