MCQ tests in Advanced Trauma Life Support (ATLS©): Development and revision

BackgroundIn Advanced Trauma Life Support (ATLS©) courses, multiple choice question (MCQ) tests are used to assess student's post course knowledge. As part of the ninth Edition Revision Process, existing MCQ tests were reviewed and revised by an International MCQ Revision group. The aim of this study was to evaluate the revision procedure and its effects.MethodsBased on psychometric data and evidence based guidelines for adequate MCQ item and test development, a detailed stepwise approach was determined and followed to evaluate the existing MCQs, and to guide test item revision or replacement.ResultsThe MCQ Revision group composed three new draft test versions comprising of 40 MCQs each. These were beta-tested among ATLS Instructors in various countries involved in ATLS. Psychometric analysis demonstrated that a minority of MCQ items required revision to create three equally balanced tests. After these final adjustments, a new set of three validated MCQ tests was available for use in 9th edition ATLS provider courses. Beta testing was performed using instructors but not students. The failure rate amongst students of ATLS provider courses increased significantly after introduction of the new MCQ tests.ConclusionATLS tests were revised and updated using current evidence based guidelines and psychometric analysis. Difficulty of the tests was not initially beta-tested on students. Increasing test item discrimination and quality resulted in lower test scores by students.