Generating algorithm for integer DST radixes in video coding

• A cost function comprising orthogonality metric and accuracy metric is established.
• A generic method is proposed to generate complete integer DST radixes.
• A basis superior over HEVC’s is revealed for the first time.

Recently released H.265 is the new generation video coding standard established by ITU and ISO jointly. H.265 is the first to apply discrete sine transform (DST) to transform unit in intra prediction block, with it implemented in an integer transform approach. Given that the integer transform radixes of DST are not unique, it is a key task to seek for them possessing strong de-correlation capability and simple arithmetic calculation as well. This paper presents a generic generating algorithm for integer DST transform radixes through a deep insight into the principle of integer DST transform. Subject to three essential constraints on integer transform matrix, this method establishes a cost function comprising orthogonality metric and accuracy metric with respect to entries of a 4 × 4 matrix, and then seeks transform radixes via a heuristic search strategy. Experimental results show that our method can find out a set of DST radixes after dozens of search steps. The revealed DST radixes not only cover the one used by H.265 but also include a basis superior over H.265’s in terms of orthogonality and precision.