Fast TU size decision algorithm for HEVC encoders using Bayesian theorem detection

•We statically analyze the correlation between the variance of residual coefficients and the TU splitting mode.•The correlation between variance of residual coefficients and transform size is exploited to reduce the number of candidate transform sizes for a given block.•CU property and Bayesian theorem detection are utilized to reduce TU processing complexity.

The emerging internationalhigh efficiency video coding (HEVC) uses variable block-size transform coding of the prediction error residual. The residual block is partitioned into multiple square transform units. The supported transform unit (TU) sizes are 4×4, 8×8, 16×16, and 32×32. The optimal TU size is chosen based on rate distortion optimization techniques. Variable TU sizes have a coding performance superior to the traditional fixed-block-size transform. Meanwhile, it causes high computational complexity at the encoder. In this paper, we propose an early termination algorithm of TU size decision for HEVC encoders. The proposed method uses the Bayesian decision theory and the correlation between the variance of residual coefficients and the transform size to reduce the number of candidate transform sizes for a given block. Simulation results show that the proposed algorithm can save 30–46% of transform processing complexity with negligible loss of coding efficiency.