Using generalizability theory for reliable learning assessments in pharmacy education

The value of conducting good assessment of learning is becoming an increasingly large focus in pharmacy education. Having a framework to understand learning assessments and recognizing sources of error that contribute to unreliability in measurement are initial steps toward designing more reliable learning assessments in pharmacy education. In this article, we provide a primer on generalizability theory (G-theory), a widely accepted psychometric model used within higher education and present original empirical findings applying G-theory to data from classroom and laboratory pharmacy education as examples. In example 1, we illustrate how the reliability of didactic course grades is affected by the length and number of examinations (i.e., more testing occasions). Our results show that a high level of reliability can be achieved with fewer overall numbers of questions spread out over more occasions of testing. In example 2, we demonstrate how G-theory can be used to establish the reliability of a drug information task in a laboratory-based course. Results reveal that, once again, using more occasions improves reliability of performance assessments. We discuss how the results can be used to begin revising a rater-scoring instrument to improve reliability. This G-theory framework and the worked examples provide a clear path forward for pharmacy educators to consider when developing learning assessments.