Profile Similarity Metrics as an Alternate Framework to Score Rating-Based Tests: MSCEIT Reanalyses

• Distance-based scores confound the shape, elevation, and scatter of rating profiles.
• Documents the utility of profile similarity metric scores for rating-based judgment tests.
• MSCEIT is best described by a single factor that is highly g-loaded, (r = .79).

Profile similarity metrics provide an ideal framework to score rating-based judgment tests. These tests are distinct in asking subjects to rate the relative adequacy of multiple responses for each question. Respondent scores should be based on how well an individual’s rating profile matches the answer key. However, respondents whose "rating-elevation" (tendency to give systematically low or high ratings) or "rating-scatter" (tendency to use more or less of the available scale) differs from the elevation and scatter of the scoring key can be given inappropriately low scores by distance-based algorithms. This can occur despite very high response accuracy with respect to the goal of correctly rating the relative adequacy of the response options for each question, “rating shape.” Using this framework, reanalyses of MSCEIT data showed that after correcting scores for elevation and scatter effects, the MSCEIT is best described by a single factor that is highly g-loaded, (r = .79), not multiple factors with low g-loadings as hypothesized by its measurement model. These results demonstrate the importance of using profile similarity metrics (i.e., shape, elevation, and scatter) to evaluate the construct-related validity for rating-based judgment tests.