AAM Based Features for Multiple Camera Visual Speech Recognition in Car Environment

This paper presents an Active Appearance Model (AAM) based multiple camera visual speech recognition (VSR) method using the shape and appearance information extracted from jaw and lip region to enhance the performance in mobile car cabin environments. Consideration of visual features along with traditional acoustic features have been found to be promising in complex auditory environment. Furthermore recent researches on multiple camera fusion to take care of pose information have shown promising result over single camera only. In this work a series of visual speech recognition experiments are carried out to study the influence of side and central faced camera on multistream visual speech recognizer. To have better information on visual articulators(Lip, Jaw etc) shape and texture model is built to extract the visual feature. Four camera audio visual corpus AVICAR is used in this research work. Individual camera streams are fused to have four stream synchronous Hidden Markov Model (SHMM) visual speech recognizer. The performance of the visual speech recognizer is improved by analyzing the relative impact of central frontal cameras with respect to side frontal cameras. Significant improvement is found compared to conventional Discrete Cosine Based(DCT) based visual features.