Spatiotemporal texture synthesis and region-based motion compensation for video compression

• Description of a complete texture analysis/synthesis.
• Rigid motions are interpolated using an affine model.
• Inter frame adapted synthesis is used for non-rigid texture regions.
• The framework has been validated within an H.264/MPEG4-AVC video codec.
• The method can be coupled with the future HEVC.

In this paper, a content-based approach for video compression is proposed. The main novelty relies on the complete texture analysis/synthesis framework, which enables the use of multiple algorithms, depending on texture characteristics. The idea comes from the efficient MPEG prediction based on a best mode selection. Existing synthesis algorithms cannot be efficient in synthesizing every kind of texture but a certain range of them.This approach is designed to be jointly used with current and future standard compression schemes. At encoder side, texture analysis includes segmentation and characterization tools, in order to localize candidate regions for synthesis: motion compensation or texture synthesis. The corresponding areas are not encoded. The decoder fills them using texture synthesis. The remaining regions in images are classically encoded. They can potentially serve as input for texture synthesis. The chosen tools are developed and adapted in order to ensure the coherency of the whole scheme. Thus, a texture characterization step provides required parameters to the texture synthesizer. Two texture synthesizers, including a pixel-based and a patch-based approach, are used on different types of texture, complementing each other.The scheme is coupled with a motion estimator in order to segment coherent regions and to interpolate rigid motions using an affine model. Inter frame adapted synthesis is therefore used for non-rigid texture regions.The framework has been validated within an H.264/MPEG4-AVC video codec. Experimental results show significant bit-rate saving at similar visual quality levels, assessed using subjective tests. The method can be coupled with the future HEVC in which blocks can be skipped by the encoder to be synthesized at decoder side.