Evaluation of objective structured clinical examination for advanced orthodontic education 12Â years after introduction

•The OSCE can help examine student learning and curriculum changes.•Significant learning appears to occur during all years of a 3-year program.•Cross-sectional and longitudinal analyses may not be complementary.•Discriminating and predictive questions can be from different domains of skills.•Both difficult and easy questions can be discriminating and predictive.

IntroductionAdvanced education programs in orthodontics must ensure student competency in clinical skills. An objective structure clinical examination has been used in 1 program for over a decade. The results were analyzed cross-sectionally and longitudinally to provide insights regarding the achievement of competency, student growth, question difficulty, question discrimination, and question predictive ability.MethodsIn this study, we analyzed 218 (82 first-year, 68 second-year, and 68 third-year classes) scores of each station from 85 orthodontic students. The grades originated from 13 stations and were collected anonymously for 12 consecutive years during the first 2 decades of the 2000s. The stations tested knowledge and skills regarding dental relationships, analyzing a cephalometric tracing, performing a diagnostic skill, identifying cephalometric points, bracket placement, placing first-order and second-order bends, forming a loop, placing accentuated third-order bends, identifying problems and planning mixed dentition treatment, identifying problems and planning adolescent dentition treatment, identifying problems and planning nongrowing skeletal treatment, superimposing cephalometric tracings, and interpreting cephalometric superimpositions. Results were evaluated using multivariate analysis of variance, chi-square tests, and latent growth analysis.ResultsThe multivariate analysis of variance showed that all stations except 3 (analyzing a cephalometric tracing, forming a loop, and identifying cephalometric points) had significantly lower mean scores for the first-year student class than the second- and third-year classes (P <0.028); scores between the second- and third-year student classes were not significantly different (P >0.108). The chi-square analysis of the distribution of the number of noncompetent item responses decreased from the first to the second years (P <0.0003), from the second to the third years (P <0.0042), and from the first to the third years (P <0.00003). The latent growth analysis showed a wide range of difficulty and discrimination between questions. It also showed continuous growth for some areas and the ability of 6 questions to predict competency at greater than the 80% level.ConclusionsObjective structure clinical examinations can provide a method of evaluating student performance and curriculum impact over time, but cross-sectional and longitudinal analyses of the results may not be complementary. Significant learning appears to occur during all years of a 3-year program. Valuable questions were both easy and difficult, discriminating and not discriminating, and came from all domains: diagnostic, technical, and evaluation/synthesis.