Phoneme-level articulatory animation in pronunciation training

Speech visualization is extended to use animated talking heads for computer assisted pronunciation training. In this paper, we design a data-driven 3D talking head system for articulatory animations with synthesized articulator dynamics at the phoneme level. A database of AG500 EMA-recordings of three-dimensional articulatory movements is proposed to explore the distinctions of producing the sounds. Visual synthesis methods are then investigated, including a phoneme-based articulatory model with a modified blending method. A commonly used HMM-based synthesis is also performed with a Maximum Likelihood Parameter Generation algorithm for smoothing. The 3D articulators are then controlled by synthesized articulatory movements, to illustrate both internal and external motions. Experimental results have shown the performances of visual synthesis methods by root mean square errors. A perception test is then presented to evaluate the 3D animations, where a word identification accuracy is 91.6% among 286 tests, and an average realism score is 3.5 (1 = bad to 5 = excellent).

► We design a data-driven 3D talking head system for articulatory animations at the phoneme level.
► We build a database of 3D articulatory movements to explore the distinctions among phonemes.
► Two visual synthesis methods are then investigated to generate the articulatory movements.
► The 3D articulators are then controlled by synthesized articulatory movements.
► We illustrate both internal and external articulatory motions for minimal pairs.