Automatic creation of scenarios for evaluating spoken dialogue systems via user-simulation

This paper proposes a novel technique to create scenarios that can be used by a user simulator for exhaustively evaluating spoken dialogue systems. The scenarios are automatically created from simple scenario-templates that the systems' developers create manually employing their knowledge about typical goals of the system's users. The scenarios contain goals, which the user simulator will try to achieve through the interaction with the systems. The goals are represented in the form of semantic frames, which are associated with user utterances of sentences and are taken from utterance corpora. In this way, the scenarios enable speech-based interaction between the simulator and the spoken dialogue systems to be evaluated. Experiments have been carried out employing two spoken dialogue systems (Saplen and Viajero), a user simulator and two utterance corpora previously collected for two different application domains: fast-food ordering and bus travel information. Experimental results show that the technique has been useful for exhaustively evaluating the systems and finding out problems in their performance that must addressed to improve them. Some of these problems are caused by acoustic similarity between some uttered words and strong speaker accents. Thus, we think these problems would have been difficult to uncover employing the user simulation techniques typically used nowadays, as they do not employ real speech and just consider semantics of user intentions.