Card sorting to evaluate the robustness of the information architecture of a protocol website

• We apply card sorting to evaluate a protocol website’s structure, after one year of use.
• Qualitative analysis of labels and cluster analysis of piles of cards were combined.
• The results offer insight into the target groups’ mental models and te everyday information needs.
• The overall website structure is validated, but additional menu categories did emerge.
• Card sorting is useful to evaluate information architecture and evolving user information needs.

ObjectivesA website on Methicillin-Resistant Staphylococcus Aureus, MRSA-net, was developed for Health Care Workers (HCWs) and the general public, in German and in Dutch. The website’s content was based on existing protocols and its structure was based on a card sort study. A Human Centered Design approach was applied to ensure a match between user and technology. In the current study we assess whether the website’s structure still matches user needs, again via a card sort study.MethodsAn open card sort study was conducted. Randomly drawn samples of 100 on-site search queries as they were entered on the MRSA-net website (during one year of use) were used as card input. In individual sessions, the cards were sorted by each participant (18 German and 10 Dutch HCWs, and 10 German and 10 Dutch members of the general public) into piles that were meaningful to them. Each participant provided a label for every pile of cards they created. Cluster analysis was performed on the resulting sorts, creating an overview of clusters of items placed together in one pile most frequently. In addition, pile labels were qualitatively analyzed to identify the participants' mental models.ResultsCluster analysis confirmed existing categories and revealed new themes emerging from the search query samples, such as financial issues and consequences for the patient. Even though MRSA-net addresses these topics, they are not prominently covered in the menu structure. The label analysis shows that 7 of a total of 44 MRSA-net categories were not reproduced by the participants. Additional themes such as information on other pathogens and categories such as legal issues emerged.ConclusionsThis study shows that the card sort performed to create MRSA-net resulted in overall long-lasting structure and categories. New categories were identified, indicating that additional information needs emerged. Therefore, evaluating website structure should be a recurrent activity. Card sorting with ecological data as input for the cards is useful to identify changes in needs and mental models. By combining qualitative and quantitative analysis we gained insight into additional information needed by the target group, including their view on the domain and related themes. The results show differences between the four user groups in their sorts, which can mostly be explained by the groups’ background. These findings confirm that HCD is a valuable approach to tailor information to the target group.