Assessing clinical judgment using the Script Concordance test: the importance of using specialty-specific experts to develop the scoring key

BackgroundThe Script Concordance test (SCT) assesses clinical judgment. The purpose of this study was to determine whether a specialty-specific scoring key improves the validity of the SCT.MethodsThirty experts from 6 general surgery disciplines answered questions pertaining to their area of expertise. We created a scoring key of 5 amalgamated expert panel members. The answers of 227 general surgery residents were analyzed.ResultsThe optimized test had a reliability level (Cronbach α) of .81. Scores increased progressively throughout all levels of training, with R5s scoring higher than R4s (R1, 42.7 ± 7.1; R2, 47.6 ± 7.5; R3, 48.7 ± 6.7; R4, 49.8 ± 7.7; R5, 52.9 ± 9.3). The average score of juniors (R1s + R2s, 45.1 ± 7.6) was significantly lower (P < .001) than seniors (R3s + R4s + R5s, 50.4 ± 8.0).ConclusionsWe showed that specialty-specific experts must be used to develop the scoring key. This has important implications in the application of the SCT on a wider level.