Concept-to-Speech generation with knowledge sharing for acoustic modelling and utterance filtering

•We present two knowledge sharing approaches for CTS generation.•Syntactic features replace prosody phrasing features in HMM-based speech synthesis.•Acoustic features are used to filter synthetic utterances for one input concept.•The HMM-based acoustic model yields comparable results without prosodic phrasing.•Utterance filtering can remove inferior synthetic utterances for the input concept.

A Concept-to-Speech (CTS) system converts the conceptual representation of a sentence-to-be-spoken into speech. While some CTS systems consist of independently built text generation and Text-to-Speech (TTS) modules, the majority of the existing CTS systems enhance the connection between these two modules with a prosodic prediction module that utilizes linguistic knowledge from the text generator to predict prosodic features for TTS generation. However, knowledge embodied within the individual modules has the potential to be shared in more ways. This paper describes knowledge sharing for acoustic modelling and utterance filtering in a Mandarin CTS system. First, syntactic information generated by the text generator is propagated to a hidden Markov model (HMM) based acoustic model within the TTS module and replaces the symbolic prosodic phrasing features therein. Our experimental results show that this approach alleviates the local hard-decision problem in automatic prosodic phrasing for Mandarin CTS systems and achieves a comparable performance to the traditional approach without explicit prosodic phrasing. Second, the acoustic features of multiple synthetic utterances expressing the same input concept are utilized to evaluate the utterance candidates. With this ‘post-processing’ mechanism, our CTS system is able to filter out inferior synthetic utterances and find an acceptable candidate to express the input concept.