Cross-user analysis: Benefits of skill level comparison in usability testing

This study presents a cross-user usability test approach and analysis technique that extends beyond merely identifying the existence of a usability problem to introducing an empirical basis for identifying the type of usability problem that exists. For experimental purposes, 60 users were tested with three levels of user-competency determined by experience in using: (1) computers, and (2) the tested application. Applying the Tukey honestly significant difference (HSD) test to each test element provided statistical comparison between different experience levels. Analysis results between experience levels suggested which levels encountered usability problems. The authors demonstrate that statistical calculations of cross-user data can render empirical support for categorizing usability problems.