A novel total variation based frame layer rate control algorithm for H.264/AVC

• Innovatively introduce total variation (TV) measure in image processing to video compression.
• Proposes a new TV distortion measure to describe video encoding quality.
• Propose a new DTV–Qstep model for interframes based on TV distortion measure.
• Present an analytic TV measure model to calculate Qstep for the initial frame.
• Propose a new incomplete derivative PID (IDPID) buffer controller.

Rate control (RC) plays a crucial role in controlling compression bitrates and encoding qualities for networked video applications. In this research, we propose a new total variation (TV) based frame layer rate control algorithm for H.264/AVC. One of its novelties is that a total variation measure, used in image processing field, is proposed to describe encoding distortion in video compression. For intraframes, we present a TV distortion–quantization (DTV–Qstep) model to obtain accurate QP step size (Qstep). Using TV measure to represent frame complexity, we also present an analytic model to calculate Qstep for the initial frame, and develop an effective scene change detection method. In addition, an incomplete derivative proportional integral derivative (IDPID) buffer controller is proposed to reduce the deviation between the current buffer fullness and the target buffer fullness, and minimizes the buffer overflow or underflow. Extensive experimental results show that, compared with JVT-W042, the proposed algorithm successfully achieves more accurate target bit rates, reduces frame skipping, decreases quality fluctuation and improves the overall coding quality.