A methodological note on ordered Q-Sort ratings

•Response patterns for Q-Sort and Likert-type ratings are compared.•Item order effects are found in Q-Sort but not in Likert-type ratings.•Items sorted later have less variance and are placed closer to the midpoint.•Item order effects attenuate some relationships with dependent measures.•Randomizing item presentation in Q-Sort ratings is recommended in some situations.

Patterns of ratings using the Q-Sort method and the Likert-type method are compared. Ordering effects are found in Q-Sort ratings that are not present in Likert-type ratings. Specifically, item order is related to both item variance and item placement, such that items appearing near the end of the Q-Sort have less variance and more central placement. This finding is verified across three measures in several datasets spanning nearly 20 years of research. Such item order effects appear to attenuate average absolute relationships (covariances and correlations) between items appearing near the end of the Q-Sort and other measures. Randomization of items may be (in some situations) a viable course of action to minimize these effects at a sample level.