Advancing Methods for Reliably Assessing Motivational Interviewing Fidelity using the Motivational Interviewing Skills Code

•We used the MISC 2.1 to rate motivational interviewing from 5 alcohol and drug use trials•We compared 3 methods of utterance-based coding reliability to traditional reliability estimates•Session-level ICCs were higher than reliability using utterance-based codes•Typical methods for assessing MISC reliability may be biased

The current paper presents novel methods for collecting MISC data and accurately assessing reliability of behavior codes at the level of the utterance. The MISC 2.1 was used to rate MI interviews from five randomized trials targeting alcohol and drug use. Sessions were coded at the utterance-level. Utterance-based coding reliability was estimated using three methods and compared to traditional reliability estimates of session tallies. Session-level reliability was generally higher compared to reliability using utterance-based codes, suggesting that typical methods for MISC reliability may be biased. These novel methods in MI fidelity data collection and reliability assessment provided rich data for therapist feedback and further analyses. Beyond implications for fidelity coding, utterance-level coding schemes may elucidate important elements in the counselor–client interaction that could inform theories of change and the practice of MI.