| Article ID | Journal | Published Year | Pages | File Type |
|---|---|---|---|---|
| 534697 | Pattern Recognition Letters | 2011 | 7 Pages |
This paper presents a novel Lip synchronization technique which investigates the correlation between the speech and lips movements. First, the speech signal is represented as a nonlinear time-varying model which involves a sum of AM–FM signals. Each of these signals is employed to model a single Formant frequency. The model is realized using Taylor series expansion in a way which provides the relationship between the lip shape (width and height) w.r.t. the speech amplitude and instantaneous frequency. Using lips width and height, a semi-speech signal is generated and correlated with the original speech signal over a span of delays then the delay between the speech and the video is estimated. Using real and noisy data from the VidTimit and in-house diastases, the proposed method was able to estimate small delays of 0.01–0.1 s in the case of noise-less and noisy signals respectively with a maximum absolute error of 0.0022 s.
Research highlights► A novel approach is proposed for the task of lip synchronization. ► The considered model and the derived analysis are consistent with the results. ► The method is able to correct for lip synchronization problems with minimal errors. ► In non-frontal faces, the method was able to achieve acceptable results. ► The method does not depend on any audio or video pilot signals. ► The method is blind to the type of the spoken language.
