A quantitative approach to conceptual, procedural and pragmatic meaning: Evidence from inter-annotator agreement

- The conceptual/procedural distinction requires testable and objective features.
- Inter-annotator agreement points to types of meaning.
- Requirements that assure the reliability of the data are provided.
- A scale for interpreting indicative values of the kappa-like coefficients is given.

This study deals with three key notions in the relevance-theoretic framework: conceptual and procedural linguistically encoded meaning on the one hand, and pragmatic meaning on the other hand. I argue that having objective and quantitative measures for distinguishing among these types of meaning is necessary. Concretely, a quantitative measure is proposed based on offline annotation experiments made by untrained native speakers. This is inter-annotator agreement measured with chance-corrected agreement coefficients, such as Cohen's kappa coefficient. In order to reliably use the three layered scale for interpreting the values of the kappa coefficient, a series of requirements regarding the building and the running of the experiment, as well as the analysis of results, must be adhered to. In this paper, the measure is applied to verbal tenses in order to identify and investigate their contextual usages. It is shown that when speakers are asked to consciously evaluate the contribution of verbal tenses to the interpretative process, three patterns emerge systematically. The first is the easiness of the task and the high rate of inter-annotator agreement when they deal with the distinction past/non-past. The second is a greater difficulty of the task and lower rates when they deal with temporal ordering eventualities. The third is the impossibility to have inter-annotator agreement beyond chance level when they have to consciously identify a subjective or non-subjective point of perspective. It is argued that this observed difference may be explained in terms of the different contents that the addressee deals with: conceptual, procedural, and respectively, purely pragmatic.