Comparison of expert-based and empirical evaluation methodologies in the case of a CBL environment: the “Orestis” experience

This paper compares several interface evaluation methods applied in the case of a computer based learning (CBL) environment, during a longitudinal study performed in three European countries, Greece, Germany, and Holland, and within the framework of an EC funded Leonardo da Vinci program. The paper firstly considers the particularities of the CBL environments presented in comparison to Distance Learning environments, as well as other educational pieces. Subsequently, the software under evaluation is described, as well as the underlying educational theories and the proposed instructional methodology regarding its effective application. In the case under study, a combination of expert-based and empirical (user-based) approaches are dealt with, and the “ideal” combination, for the consumption of the fewest possible resources in order to achieve the maximum outcome for the assessment of the software is investigated. A combinatory evaluation approach is argued for the provision of the best results in the case of complex environments such as educational ones. The results of this study show that both approaches, expert-based and empirical, have been found to perform adequately, although the empirical methodologies are always preferable. Therefore, this study concludes by proposing a combination of an expert-based approach during the early stages of the design cycle with an observational evaluation, performed by the instructors and/or the designers, which has to provide the best cost/performance factor.