Modeling input modality choice in mobile graphical and speech interfaces

• Decreased input performance of a specific modality, decreases the usage of this modality.
• Decreased efficiency of the GUI, increases VUI usage.
• A model predicting modality choice is proposed.
• The model can be deployed for automatic usability evaluation.
• System designers are provided with working knowledge about expected modality usage.

In this paper, we review three experiments with a mobile application that integrates graphical input with a touch-screen and a speech interface and develop a model for input modality choice in multimodal interaction. The model aims to enable simulation of multimodal human–computer interaction for automatic usability evaluation. The experimental results indicate that modality efficiency and input performance are important moderators of modality choice. Accordingly, we establish a utility-driven model that provides probability estimations of modality usage, based on the parameters of modality efficiency and input performance. Four variants of the model that differ in training data are fitted by means of Sequential Least Squares Programming. The analysis reveals a considerable fit regarding averaged modality usage. When applied to individual modality usage profiles, the accuracy decreases significantly. In an application example it is shown how the modality choice mechanism can be deployed for simulating interaction in the field of automatic usability evaluation. Results and possible limitations are discussed.