Hybrid model-and-object-based real-time conversational video coding

• A low bit-rate conversational video coding method is proposed.
• The face region is tracked with a face model and coded with model-based coding.
• The head-shoulder region is segmented and coded with mesh-based coding.
• The two regions are integrated; bit-allocation is performed by joint optimization.
• Rate-distortion improvements of more than 5 dB over existing methods were observed.

Bandwidth-constrained real-time conversational video communications (such as mobile teleconferencing) require video codecs with good rate-distortion characteristics at low bit-rates and modest computational complexity. While target-specific object-based and model-based coding methods have been proposed for low bit-rate conversational video coding, difficulties in generalization and high computational complexity hinder their practical utilization. In this paper, we propose a low bit-rate coding method for typical conversational video by combining two-dimensional model-based coding of face regions and object-based coding of non-face head-shoulder regions, achieving high-quality face reconstruction and low overall bit-rate with real-time encoding capability. Experiments on typical conversational test sequences confirm that, compared to other conversational video codecs, our model-and-object-based coding method offers superior rate-distortion performance at low bit-rates.