Why Statistics Matter: Limited Inter-Rater Agreement Prevents Using the Psoriasis Area and Severity Index as a Unique Determinant of Therapeutic Decision in Psoriasis

Although the demand for evidence-based decisions is increasing in clinical practice, recent systematic reviews on the accuracy of existing psoriasis severity scales, including the Psoriasis Area and Severity Index (PASI), suggest that their validity is not fully characterized. We simulated the evaluation of PASI by two practitioners in 1,000 sets of 100 patients. PASI data from several practitioners who examined the same patients were used to generate PASI scores by two practitioners, in order to compare how well commonly used statistics assess the inter-rater agreement for the PASI. Because the PASI score has an asymmetric distribution, statistics such as Pearson's linear correlation coefficient “r” and Spearman's rank correlation coefficient overestimated the inter-rater agreement of PASI as compared with the intra-class correlation coefficient (ICC; r=0.8, ρ=0.7, ICC=0.5). When restricting the analysis to patients with a PASI <20, inter-rater agreement severely decreased (r=0.38, ρ=0.41, ICC=0.17), resulting in unacceptable therapeutic decision agreement (κ=0.38). Our study indicates that owing to the skewed distribution of the PASI its validity to influence therapeutic decisions is questionable. The ICC is preferable to the commonly used statistics (r and ρ) for assessing the inter-rater agreement reliability of asymmetrically distributed scores such as the PASI.