Estimating the number of usability problems

The number of usability problems discovered in a user trial or identified in a heuristic evaluation can never be claimed to be exhaustive. This raises the question of how many usability problems remained undetected. In ergonomics/human factors research this subject matter is often addressed by asking how many participants are sufficient to discover a specific proportion of the usability problems. Current approaches to answer this question suffer from various biasing mechanisms, which undermine the credibility of the popular ‘rule of thumb’ that five participants are sufficient for the discovery of 80% of ‘all’ usability problems. This 5-user rule appears to be speculative in its application as a stop rule. In this paper, I compare actual estimates of the number of usability problems. Underestimation surfaces as a permanent threat. The so-called Turing estimate (CT) appears to be the most satisfactory. However, also CT estimates may suffer from underestimation. Therefore max(CT,CF) with the CF estimate based on partitioned frequencies is proposed as the most adequate estimate of the number of usability problems in the studies presented.