Validating Medical Students’ Self-Report of Smoking-Related Communication Skills and Educational Experiences

ObjectivePediatric clerkships can be important settings for medical students to learn how to perform smoking assessments and counseling with parents. In evaluating an educational intervention that promotes this skill, we assess whether students’ self-report of this counseling skill was a valid measure of actual behavior.MethodsA trained observer evaluated student smoking assessment and counseling practices during pediatric well-child visits at 5 clinical sites in eastern Massachusetts. The external observations of behavior were used as a gold standard, and we determined the accuracy of the students’ self-report of their smoking counseling practices with families and of their preceptors’ educational interventions.ResultsWe observed 38 pediatric preceptors and 85 Boston University School of Medicine (BUSM) III students in 85 clinical encounters. Sensitivities of the students’ report of assessing smoking practices and counseling parents and children ranged from 0.60–1.00, and specificities ranged from 0.41–0.88. Specifically, with regards to the students’ report of performing a smoking assessment with the family, the sensitivity was 0.97 (95% confidence interval, 0.89–0.99) and the specificity was 0.88 (95% confidence interval, 0.72–0.97). For measures where the observer could not determine practice, agreement between the student and preceptor was between 57% and 83%. Specifically, with regard to whether the preceptor made expectations clear with the student, students and preceptors agreed 83% of the time.ConclusionAlthough direct observations of behavior may still be the most accurate report of true practice, when this is not feasible, student self-report appears to be a valid measure of smoking assessment and counseling practices during pediatric clerkships.