Beta Test of Web-Based Virtual Patient Decision-Making Exercises for Residents Demonstrates Discriminant Validity and Learning

IntroductionCorrect clinical decision-making is a key competency of surgical trainees. The purpose of this study was to assess validity and effect on resident decision-making accuracy of web-based virtual patient case scenarios in general surgery training.Material and methodsDuring the 2013-2014 academic year, the use of web-based virtual patient scenarios for teaching and assessment of resident critical thinking and decision-making was assessed in 2 urban university−based residency programs. In all, 71 residents (PGY [postgraduate year] 1 = 21, PGY2 = 11, PGY3 = 14, PGY4 = 13, and PGY5 = 12) took the cases over the course of the academic year. Cases were made available to the residents online 1 week before a scheduled debriefing conference with a faculty facilitator and were completed by residents individually on their own schedule. At the completion of each case attempt, residents were given a computer-generated score and feedback. Residents were allowed to repeat the cases before the debriefing if they wished. Cases were required to be completed by 48 hours before the conference, at which time a faculty report was computer generated that measured group and individual performance and identified the frequency of errors in decision-making. This report was reviewed with the residents in the faculty debriefing, and teaching focused on the knowledge gaps identified in the reports.ResultsThe mean percentage of assigned cases completed by categorical residents was 85.7%. Mean score (maximum possible = 100) on the cases increased by resident year (PGY1 = 45.3, PGY2 = 49.3, PGY3 = 53.6, PGY4 = 57.5, and PGY5 = 61.8), a 25% increase between PGY1 and PGY5 (p < 0.001 by analysis of variance). In all, 45 (63%) residents chose to repeat at least 1 case before the debriefing. The number of repetitions of individuals on the same case varied from a minimum of 1 to a maximum of 5. On repeated cases, mean scores rose (attempt 1 = 22.6, attempt 2 = 69.3, attempt 3 = 72.1, attempt 4 = 77.5, attempt 5 = 100, p < 0.0001 by analysis of variance). Paired t tests on case repetition using each resident as his-her own control showed that scores rose by 46 points between attempt 1 and attempt 2 (p < 0.001).Conclusions(1) In a beta test of web-based scenarios that teach and assess clinical decision-making, resident scores improved by 25% from PGY 1 to PGY5 in a stepwise and statistically significant manner, suggesting that such exercises could serve as milestones for competency assessment. Additional studies are needed to acquire evidence for other forms of validity. (2) Repetition of cases after feedback led to highly significant increases in performance, suggesting that requiring repeated training to reach defined levels of competence is practical.