Generalized estimating equations with model selection for comparing dependent categorical agreement data

Many studies in biomedical fields are carried out using diagnoses reported by different raters to evaluate the agreement of multiple ratings. The most popular indices of agreement are kappa measures including Cohen’s kappa and weighted kappa for binary and ordinal outcomes, respectively. However, when raters assess the same observation on two or more occasions, these ratings are dependent and so the correlation between kappa estimates must be considered when making inferences. In this paper, we focus on testing the equality of correlated kappa coefficients using the generalized estimating equation (GEE) approach and applying quasi-likelihood under the independence model criterion (QIC) measures for model selection. Simulation studies are conducted to compare the performance between GEE with and without QIC measures, weighted least squares (WLS) and independence approaches for binary and ordinal data. Two applications are illustrated: a comparison of two methods for assessing cervical ectopy, and similarity in myopic status for monozygous twins and dizygous twins. We conclude that when performing the QIC model-selection procedure in GEE models and taking into account the correlation between kappa measures, it leads to nominal type I errors and larger powers.