Incorporation of Core Competency Questions Into an Annual National Self-Assessment Examination for Residents in Physical Medicine and Rehabilitation: Results and Implications

ObjectiveTo determine the performance and change over time when incorporating questions in the core competency domains of practice-based learning and improvement (PBLI), systems-based practice (SBP), and professionalism (PROF) into the national PM&R Self-Assessment Examination for Residents (SAER).DesignProspective, longitudinal analysis.SettingThe national Self-Assessment Examination for Residents (SAER) in Physical Medicine and Rehabilitation, which is administered annually.ParticipantsApproximately 1100 PM&R residents who take the examination annually.InterventionsInclusion of progressively more challenging questions in the core competency domains of PBLI, SBP, and PROF.Main Outcome MeasurementsIndividual test item level of difficulty (P value) and discrimination (point biserial index).ResultsCompared with the overall test, questions in the subtopic areas of PBLI, SBP, and PROF were relatively easier and less discriminating (correlation of resident performance on these domains compared with that on the total test). These differences became smaller during the 3-year time period. The difficulty level of the questions in each of the subtopic domains was raised during the 3 year period to a level close to the overall exam. Discrimination of the test items improved or remained stable.ConclusionsThis study demonstrates that, with careful item writing and review, multiple-choice items in the PBLI, SBP, and PROF domains can be successfully incorporated into an annual, national self-assessment examination for residents. The addition of these questions had value in assessing competency while not compromising the overall validity and reliability of the exam. It is yet to be determined if resident performance on these questions corresponds to performance on other measures of competency in the areas of PBLI, SBP, and PROF.