Performance-based usability evaluation of a safety information and alarm system

Evaluation of the appropriateness of information technical systems for complex professional usage in safety-critical contexts poses significant methodical and practical challenges. In this study, the usability of a Safety Information and Alarm Panel (SIAP) in a nuclear power plant control room was tested. An integrated validation concept was used that included a new approach to measuring system and operator performance in complex work environments. The tested system was designed to aid the operators in severe disturbance and emergency situations. It had already been implemented at a nuclear power plant. The study was conducted in a full-scope training simulator. The results verified that an acceptable level of performance could be achieved when using the SIAP. When the operators' practices were analysed by a habit-centred analysis, it was discovered that the effects of the SIAP differed between crews and between test scenarios. Thus, the SIAP tended to promote coherence of practices but reduce situatively attentive action. In diffuse task contexts the tool failed to support the shift supervisor's control of the overall process situation, his awareness of the crew's work load and his ability to update the crew's awareness of the process. The operators reported that the system supported their process control activity and reduced stress in the situation, but the shift supervisors and operators also noticed some possible negative effects of the tool. These subjective evaluations corresponded to the effects observed in practice. The results revealed the complexity of the implementation of new tools into professional practice. It was proposed that a validation project should focus on the trajectory of development of the entire distributed cognitive system instead of comprehending validation studies as tests of the effects of information systems on a pre-defined process output. Formative evaluation criteria are needed in projecting distributed cognitive systems.