Assessing the reliability, validity and acceptance of a classification scheme of usability problems (CUP)

• A thorough motivation for the need of a classification scheme of usability problems.
• A comparison of CUP with existing classification scheme using 9 parameters.
• An analysis of validity including models of relationships between CUP variables and three pertinent tasks that developers carry out.
• A discussion on the implication for further enhancement of CUP with a specific focus on proposing parameters to address the significant issue of prioritizing usability problems.
• A discussion on the downstream utility of CUP.

The aim of this study was to evaluate the Classification of Usability Problems (CUP) scheme. The goal of CUP is to classify usability problems further to give user interface developers better feedback to improve their understanding of usability problems, help them manage usability maintenance, enable them to find effective fixes for UP, and prevent such problems from reoccurring in the future. First, reliability was evaluated with raters of different levels of expertise and experience in using CUP. Second, acceptability was assessed with a questionnaire. Third, validity was assessed by developers in two field studies. An analytical comparison was also made to three other classification schemes. CUP reliability results indicated that the expertise and experience of raters are critical factors for assessing reliability consistently, especially for the more complex attributes. Validity analysis results showed that tools used by developers must be tailored to their working framework, knowledge and maturity. The acceptability study showed that practitioners are concerned with the effort spent in applying any tool. To understand developers’ work and the implications of this study two theories are presented for understanding and prioritising UP. For applying classification schemes, the implications of this study are that training and context are needed.