From an expert-driven paper guideline to a user-centred decision support system: A usability comparison study

ObjectiveTo assess whether a user-centred prototype clinical decision support system (CDSS) providing patient-specific advice better supports healthcare practitioners in terms of (a) types of usability problems detected and (b) effective and efficient retrieval of childhood cancer survivor's follow-up screening procedures compared to an expert-driven paper-based guideline.Methods and materialsA user-centred design (UCD) process was employed to design a prototype CDSS. Usability problems in information retrieval with the paper-based guideline were assessed by think-aloud analysis with 13 participants. Both simple and more complex tasks were applied. The analysis provided input for the UCD process of the prototype. The usability of the prototype CDSS was subsequently evaluated by think-aloud analysis with the same participants. Usability problems of the paper-based guideline and the prototype CDSS were compared by using the classification of usability problems scheme. In addition, efficiency (time to complete task) and effectiveness (completeness of retrieved screening procedures) of information retrieval of participants in the expert-driven paper-based guideline and the user-centred prototype CDSS were compared.ResultsUsability problems in both the paper-based guideline and the CDSS prototype were mainly classified as ‘incongruent with participants’ mental model’. The prototype CDSS reduced this type of problem from 17 to 6 problems. The time to perform simple information retrieval tasks increased by 58 s when using the prototype CDSS, however, it resulted in a 58% improvement in task completeness compared to the paper-based guideline. The time to perform complex scenarios decreased by 3:50 min with the prototype CDSS, with 17% higher completeness compared to the paper-based guideline.ConclusionAnalysis showed that usability problems experienced by healthcare practitioners when using a paper-based guideline could be overcome by implementing the guideline in a user-centred CDSS design. Although different types of usability problems were experienced with the prototype CDSS, they did not inhibit effective and efficient performance of tasks in the system. The usability problem analysis of the paper-based guideline effectively supported comparison of usability problems found in the two information retrieval systems and it supported the UCD of the CDSS.