Simplified algorithms for rate-distortion optimization in high efficiency video coding

• Analysis of the previous research for H.264/AVC RDO and its application to HEVC RDO.
• Proposing new simplified algorithms for HEVC RDO.
• Analysis of the relation between computation complexity and quality degradation for proposed algorithms.
• Hardware estimation for proposed algorithms.

HEVC is the latest coding standard to improve the coding efficiency by a factor of two over the previous H.264/AVC standard at the cost of the increased complexity of computation rate-distortion optimization (RDO) is one of the computationally demanding operations in HEVC and makes it difficult to process the HEVC compression in real time with a reasonable computing power. This paper aims to present various simplified RDO algorithms with the evaluation of their RD performance and computational complexity. The algorithms for the simplified estimation of the sum of squared error (SSE) and context-adaptive binary arithmetic coding (CABAC) proposed for H.264/AVC are reviewed and then they are applied to the simplification of HEVC RDO. By modifying the previous algorithm for H.264/AVC, a new simplified RDO algorithm is proposed for modifying the previous algorithm for H.264/AVC to be optimized for the hierarchical coding structure of HEVC. Further simplification is attempted to avoid the transforms operations in RDO. The effectiveness of the existing H.264/AVC algorithms as well as the proposed algorithms targeted for HEVC is evaluated and the trade-off relationship between the RD performance and computational complexity is presented for various simplification algorithms. Experimental results show that reasonable combinations of RDO algorithms reduce the computation by 80–85% at the sacrifice of the BD-BR by 3.46–5.93% for low-delay configuration.