Comparing the comprehensiveness of three expert inspection methodologies for detecting errors in interactive systems

• Empirical study of comprehensiveness of SHERPA, HEHAZOP and Cognitive Walkthrough (CW).
• CW found errors related to user goals and plans more readily than SHERPA and HEHAZOP.
• SHERPA and HEHAZOP both found more errors about user actions and feedback.
• Cognitive phase taxonomy allowed for more complete coding of the error types than the behavioural deviation taxonomy.
• There was very little overlap in the problems found by each method.

Expert inspection methods provide a means of evaluating interactive systems for error throughout the design lifecycle. Experts have a wide variety of methods available to them for detecting either potential user errors or usability problems that will provoke error; however, the data on what types of errors are detected by each method is very thin. This paper, presents the results of a study into the comprehensiveness of three expert inspection methods that were applied by nine evaluators across three devices. This study produced 350 errors that were analysed to identify, compare and contrast what types of errors were detected by each method. Of particular interest, the investigation revealed that a substantial number of distinct errors were detected by only one method. The paper closes with a discussion of the implications of these results on future practice for multi-method approaches as well as directions for future investigations.