Establishing Benchmarks For Minimum Competence With Dry Lab Robotic Surgery Drills

The Robotic Objective Structured Assessment of Technical Skills (R-OSATS) is a previously validated assessment tool that is used to assess 5 standardized inanimate robotic surgery drills. R-OSATS is used to evaluate performance on surgical drills, with scores of 0 to 20 for each drill. Our objective was to establish the minimum threshold score that denotes competence on these drills. Thus, we performed a standard setting study using data from surgeons and trainees in 8 academic medical centers. Cutoff scores for the minimal level of competence using R-OSATS were established using 2 techniques: the modified Angoff and the contrasting groups methods. For the modified Angoff method, 8 content experts met and, in an iterative process, derived the scores that a minimally competent trainee should receive. After 2 iterative rounds of scoring and discussion with the modified Angoff method, we established a minimum competence score per drill with high agreement (rWG range, 0.92-0.98). There was unanimous consensus that a trainee needs to achieve competence on each independent drill. A second method, the contrasting groups method, was used to verify our results. In this method, we compared R-OSATS scores from “inexperienced” (34 postgraduate year 1 and 2 trainees) with “experienced” (22 faculty and fellow) robotic surgeons. The distributions of scores from both groups were plotted, and a cutoff score for each drill was determined from the intersection of the 2 curves. Using this method, the minimum score for competence would be 14 per drill, which is slightly more stringent but confirms the results obtained from the modified Angoff approach. In conclusion, using 2 well-described standard setting techniques, we have established minimum benchmarks designating trainee competence for 5 dry lab robotic surgery drills.