Examining validity evidence for multidimensional forced choice measures with different scoring approaches

This study aims to investigate whether the Thurstonian Item Response Theory (TIRT), a recently developed normative scoring method, yields better validity evidence than the traditional partially ipsative scoring methods for multidimensional forced choice measures. For this purpose, we compared the construct- and criterion-related validity evidence of three different scoring methods for MFC measures, including 1) a partially ipsative method based on classical test theory (PI-CTT), 2) an analogous partially ipsative method using the graded item response theory (PI-IRT), and 3) the TIRT method. We also included in our analyses the validity evidence for a single-statement (SS) Big Five personality measures. Overall, the validity evidence for the three types of MFC scoring methods was comparable to the validity evidence for the traditional summative scoring (SS-CTT) method. Interestingly enough, the PI-CTT scoring method, the simplest method for MFC scoring, was about as effective as the more complex TIRT method. We discuss practical implications of the results and offer suggestions for future research.