When proportion consensus scoring works

Most objectively scored tests use items with easily identifiable correct answers. When such veridical scoring keys cannot be constructed, researchers sometimes use proportion consensus scoring (PCS) to identify the best answers. To determine if PCS identifies the best answers, we scored a test using both PCS and veridical scoring. Among 353 undergraduates, regular PCS, two-stage PCS, and expert PCS all had high correlations for easy items, but no PCS methods had high correlations for difficult items. Thus, PCS cannot reliability identify the best answers to individual items. However, PCS worked well for total scores. For easy items, total scores had correlations above .99 for all PCS methods. For difficult items, expert and two-stage PCS had correlations of .92 and .82 for the 60-item test. Thus, expert and two-stage PCS can be justified (even for difficult items) if the scoring key is based upon people who truly possess some degree of expertise and if scores are summed over many items.

► A proportion consensus score is the proportion of norm group who gave that response.
► PCS reliably identifies best answers for easy/moderate items but not difficult ones.
► Aggregating over many items, total PCS scores correlate highly with veridical scores.
► Expert PCS works better than two-stage PCS, which works better than regular PCS.
► We conclude PCS can be justified if items are easy/moderate or many items are used.