Assessment of the validity and intrusiveness of online-probe questions for situation awareness in a simulated air-traffic-management task with student air-traffic controllers

Online-probe techniques for measuring situation awareness (SA) represent an alternative to offline-probe methods in which operators are queried about the situation during scenario freezes with displays blanked. Online-probe queries are administered while the task is ongoing and displays active. However, online-probes techniques have not been validated to the same extent as offline probes, and have been criticized because asking questions about the current or future situation while the task is active could change the operator’s subsequent awareness of the situation. The present investigation examined the possibility of the intrusiveness of the online-probe technique to an operator’s SA. Twelve student air-traffic controllers (ATCos) served as participants and managed traffic in six 30-min scenarios in which online probes were administered regularly. Off-nominal flight-plan deviations followed some probe queries. Three pre-event question types were administered prior to scheduled flight-plan deviations. These pre-event questions were either relevant to the deviating event and subsequent conflict, relevant to conflicts but not the deviating event, or unrelated to both conflicts and the deviating event. The type of pre-event question preceding a flight-plan deviation did not change the time to detect the deviating event nor the number of losses of separation (LOS) created by the deviation. Moreover, online probes were related to measures of sector safety.

► Examined intrusiveness of online-probe technique on student ATCos’ SA.
► Three question types were administered prior to scheduled flight-plan deviations.
► Detection times and number of LOS for off-nominal events were analyzed.
► Neither were changed by the relationship of questions to off-nominal events.