The effect of response complexity and media on user restatement with multimodal virtual assistants

We investigate the occurrence of user restatement when there is no apparent error in Intelligent Virtual Assistant (IVA) understanding in a multimodal customer service environment. Analysis was performed on 2998 sessions with a live customer service IVA deployed on a major airline company website and mobile application. Restatements of a user's previous turn in a conversation are separated into two classes by evidence of dissatisfaction in the IVA response. We consider combinations of response media and linguistic complexity features and determine through detailed statistical analysis which combinations of features should be minimized to improve user comprehension. A discussion of these findings follows demonstrating how the presentation of response in addition to the formulation of response text in a multimodal environment can have an effect on user understanding. Through this analysis, 1 we derive guidelines when crafting responses for designers of multimodal IVAs.