An efficient intra-mode selection algorithm for H.264 based on edge classification and rate-distortion estimation

The H.264/AVC is the newest video coding standard recommended by ITU-T and MPEG. Compared with all existing video coding standards, H.264 can achieve superior performance by using many advanced techniques. Intra mode selection is an important feature in H.264 standard and can reduce the spatial redundancy in intra-frame significantly. An efficient rate distortion optimization (RDO) technique is employed in H.264 to choose the best mode for each MB, but the computational cost increases drastically. In this paper, a fast intra-mode selection algorithm is introduced. By using a fast edge detection method which is based on non-normalized Haar transform (NHT), edge for each sub-block can be extracted. Based on the local edge information, only few intra-modes are chosen as mode candidates. A fast RDO algorithm is also proposed in this paper which is based on accurate rate-distortion estimation model and the fast intra-mode RDO method. By combining these two methods, computational load is reduced remarkably. Experimental results show that this fast intra-mode selection scheme can lessen the encoding time significantly with little loss of bit-rate and visual quality.