A novel scanning pattern for entropy coding under non-square quadtree transform (NSQT)

High-efficiency video coding (HEVC) introduces a non-square quadtree transform (NSQT), which has brought about higher coding efficiency than square quadtree transform. However, the scanning pattern does not fit the energy distribution of the quantization coefficients well enough, which would worsen the entropy coding efficiency. In order to fit the energy distribution better, a novel scanning pattern is proposed in this paper. The spatial frequency position distribution of the non-square transformed block is derived. Based on the distribution, the energy distribution of the non-square transformed bock is analyzed in theory and tested in experiment. It is found that the energy decreases from upper left corner to lower right corner and the energy in lower left corner is close to that in upper right corner. Based on this energy distribution, the scanning pattern for the quantization coefficients in the non-square matrix is rebuilt to fit the entropy coding, which prefers a scanning pattern in ascending order in the frequency domain. Simulation results show that the proposed scanning pattern achieves on average 2.1% BD-rate saving and 9.05–13.19% bitrates saving for the non-square transformed blocks with negligible loss in coding computational complexity and peak signal-to-noise ratio (PSNR), compared to the present scanning pattern in NSQT.