Mode dependent loop filter for intra prediction coding in H.264/AVC

• We analyze the limitations of the deblocking loop filter (DLF) in H.264/AVC.
• We investigate the intra modes combinations characteristics in H.264/AVC.
• A novel adaptive loop filter was proposed by combining a bank of wiener filters.
• The filter parameters are trained for different intra modes combinations.

In this paper, a high performance and low complexity loop filter is proposed for intra prediction coding. Although the deblocking loop filter (DLF) has achieved outstanding performance on suppressing quantization noise, it also induces details information loss because of the smoothing operation. To achieve better restoration performance, we propose a filter set named mode dependent loop filter (MDLF) which adaptively select the filter coefficients according to various local characteristics. In the homogeneous areas, the task of the filter emphasizes on smoothing the noise. In the heterogeneous areas, the proposed filter concentrates on preserving the details. Based on the spatial correlation assumption and statistical analysis, the intra mode combination is used to classify the training samples with different local characteristics. Then the classical least mean square error framework is employed to solve the coefficients for the proposed filter set. In this way, a more efficient adaptive loop filter scheme can be achieved for specific intra mode combination. Experiment results show that the proposed loop filter achieves superior coding gains compared to the H.264/AVC High Profile. Furthermore, relative to QALF+DLF, a comparable performance also can be achieved by the proposed MDLF with far less complexity increase.