Using subject specialists to validate an ESP rating scale: The case of the International Civil Aviation Organization (ICAO) rating scale

• The study presents a validation of the ICAO language proficiency requirements.
• The judgments of pilots and language trained raters are compared.
• Implications for rating scale design for LSP assessment are discussed.

As part of the English-language proficiency requirements for pilots and air traffic controllers, the International Civil Aviation Organization (ICAO) published a rating scale designed to assess pilots’ and air traffic controllers’ aviation English proficiency. However, it is not clear how this scale was developed. As part of an attempt to address the need for validation, this paper presents a study involving focus group interviews with pilots. Ten pilots listened to performances of test takers taking a variety of aviation English tests. The pilots were asked to rate the acceptability of the pilot’s language for (a) communicating with other pilots and (b) radiotelephony communications with air traffic control. The focus groups had two aims: (1) to establish the ‘indigenous’ assessment criteria pilots use when assessing the language ability of peers and (2) to establish what level is sufficient as the operational level. The results showed that the pilots focused on some but not all of the criteria on the ICAO scale. Whilst listening to the performances, they also often focused on the speakers’ technical knowledge. The paper proposes a model of how industry professionals can be involved in the validation of an LSP rating scales.