Effects of training intensity on observers’ ratings of anxiety, social skills, and alcohol-specific coping skills

Few studies have addressed the amount of training needed to obtain reliable ratings in behavioral observation data. The current study examined the effects of differing intensities of frame-of-reference (FOR) rater training on observers’ ratings of anxiety, social skills, and alcohol-specific coping skills in community volunteers with and without social anxiety and alcohol use disorders. Interrater reliability was assessed by comparing three training conditions (no-training, moderate FOR, and intensive FOR) on discrepancies between raters’scores and the strength of association between raters’scoring patterns. The discrepancies between raters’ scores were significantly larger in the control condition than in the intensive and moderate training conditions. Generally, small and nonsignificant differences were found between intensive and moderate training's discrepancy scores. Strength of association results showed significantly lower correlations in the control group compared to the intensive group. However, these correlational results showed less consistent differences between the moderate and other training conditions; differences when found were in the expected directions. Study findings suggest that differing training intensities can affect rating scores and that interrater reliability may be meaningfully assessed through multiple methods.