Reading paths and visual perception in multimodal research, psychology and brain sciences

This paper argues that the concept of a reading path in multimodal research can be improved by previous research on visual perception in psychology and brain sciences, and particularly by the work done within eye-tracking studies. The paper argues that in its current state, the concept of a reading path is not sufficiently reliable due to the lack of empirical testing and therefore presents a methodological proposal to improve the current situation.Thus, the paper identifies common areas of interest related to visual perception, where the research interests of the disciplines meet and enable reciprocal input. It is suggested that multimodal research is capable of describing the high-level factors that affect visual perception, whereas eye-tracking equipment can track the actual reader behaviour. Applicable state-of-the-art theories of multimodal analysis are then described, along with the technological requirements for the eye tracker and its software.XML annotation, output and transformations are proposed for combining the results of multimodal analysis and the observer behaviour captured using an eye tracker. Finally, the paper presents a hypothesis on the relationship of visual perception and multimodal semiosis, which may be evaluated using the proposed method combining multimodal analysis and eye-tracking.

► The concept of a ‘reading path’ in multimodal research is contested by research in psychology and brain sciences. ► Multimodal research may describe certain high-level factors of visual perception. ► Methods of multimodal analysis and eye-tracking studies can be interfaced via XML.