A Measure of Ordinal Concordance for the Evaluation of University Courses

In this paper we apply the s* statistic, aimed to measure the inter-rater agreement between observers in case of ordinal variables, to the evaluation of the quality of University courses. The objective is to measure the inter-rater agreement between students, along with their satisfaction, in order to verify the consistency of judgments expressed by independent observers. s* is a modification of a previously proposed index, which avoids the problem of paradoxes of Cohen's and Fleiss’ kappa statistics. We present the s* index from both a descriptive and an inferential point of view. In particular, as far as statistical inference is concerned, we show that s* is a biased estimator of the inter-rater agreement in the population and, under the null hypothesis of inter-rater agreement by chance, s* is asymptotically normally distributed.