The usability inspection performance of work-domain experts: An empirical study

It is a challenge for usability experts to perform usability inspections of interactive systems that are tailored to work-domains of which these experts have little knowledge. To counter this, usability inspections with work-domain experts have been explored, but little empirical research has been reported on these experts' performance as evaluators. The present study compared the performance of work-domain experts and usability experts with respect to validity and thoroughness. The work-domain experts were characterized by high computer experience and low system experience. The usability experts were recruited from different ICT companies. The usability inspection method applied was group-based expert walkthrough; a method particularly developed to support non-usability experts as evaluators. The criterion for performance comparison was established through user tests. Fifteen work-domain experts and 12 usability experts participated in the study. The work-domain experts generated equally valid but less thorough usability inspection results than did the usability experts. This finding implies that work-domain experts may be used as evaluators in usability inspections without compromising validity. Moreover, the usability inspection performance of nominal groups of evaluators was explored. It was found that nominal groups of work-domain experts produced results of similar quality as did nominal groups of usability experts, given that group size is disregarded. This finding may be used as basis for hypotheses in future studies on the usability inspection performance of nominal groups of work-domain experts.