In the eye of the beholder? An examination of the inter-rater reliability of the LSI-R and YLS/CMI in a correctional agency

• Inter-rater reliability is an often overlooked psychometric property of risk assessment instruments in the criminal justice system
• We conduct inter-rater reliability studies on the Level of Service Inventory-Revised (LSI-R) and Youth Level of Service/Case Management Inventory (YLS/CMI) in a correctional agency
• Findings indicate adequate reliability for the LSI-R and excellent reliability for the YLS/CMI

PurposeTo examine the inter-rater reliability of two risk assessment tools: The Level of Service Inventory-Revised (LSI-R) and the Youth Level of Service/Case Management Inventory (YLS/CMI).MethodsTwo identical experiments are reported. For both studies, a random sample of 10 offenders were interviewed and videotaped with each tool (totaling 20 offenders). The tapes were then shown to a random selection of 20 raters (for a total of 40 raters) employed at a state agency. The fully-crossed design allowed each of the raters to rate the each of the cases, resulting in 200 total risk score observations for each tool. Inter-rater reliability analyses were then conducted.ResultsThe LSI-R demonstrated adequate to fair reliability, with certain domains showing lower reliability. Overall, the LSI-R had an ICC of .65. The YLS/CMI demonstrated higher reliability (ICC of .78). In addition, for the LSI-R study, comparisons were made between staff raters who work in a facility versus those in the community (e.g., probation officers). For the YLS/CMI study, comparisons were made between incarcerated offenders versus probationers. Neither comparison yielded consistent differences.ConclusionsThe YLS/CMI is generally reliable. The LSI-R showed less reliability. However, each study showed certain domains with less than ideal reliability.