Measure for Measure: A two study multi-trait multi-method investigation of construct validity in IS research

Given the importance and criticality of instrument validation in IS research, the main objective of this study is to provide a systematic assessment of IS construct validity via multi-trait multi-method (MTMM) matrix. To do so, the paper uses structurally different methods – neurophysiological and self-reported instruments – to measure three important and commonly used IS constructs: engagement, arousal and cognitive load in two different experimental settings. The experiments involved seventeen (17) and twenty-four (24) participants respectively and consisted in using different IS to execute a set of both instrumental and hedonic tasks. The results generally support MTMM matrix expectations and shed light on the complexity of detecting the nature of mono-method bias. Specifically, the results show that primitive perceptual IS constructs such as arousal seem to be less affected by mono-method bias, whereas more complex perceptual constructs such as engagement or cognitive load have higher within method correlations. There are two complementary explanations for the within method correlations: (a) a combination between complexity of trait and method and (b) method effects that are congeneric.

• Results generally support a correspondence between self-reported and neurophysiological measures. • Primitive perceptual constructs are not affected by mono-method bias. • Complex perceptual measures are more likely to suffer from mono-method bias. • Within method correlations can be due to complexity of trait method and congeneric method effects.