A novel distortion criterion of rate-distortion optimization for depth map coding

In 3D video coding systems, depth maps are not displayed to the viewers, but provide the geometric information to generate virtual views. To ensure the quality of virtual views, the rate-distortion optimization (RDO) in depth map coding adopts the virtual view distortion as the distortion item. The virtual view distortion comes from the reconstructed color video distortion and depth distortion. It is usually recognized that the virtual view distortion caused by reconstructed color video distortion is independent of that in depth map coding. Preliminary experiments reveal that the virtual view distortion in depth map coding is also influenced by the reconstructed color video distortion. Therefore, we proposed a novel distortion criterion of depth map coding in which the reconstructed color video distortion is modeled and joins into the virtual view distortion calculation. Correspondingly, the associated Lagrange multiplier is also proposed. Experimental results demonstrate that the method by integrating the proposed distortion criterion into RDO for depth map coding can achieve an average 12.72% bitrate saving compared with SSD based RDO method and can also lead a bitrate reduction (0.64%) compared with the existing distortion estimation method in the current 3D-HEVC reference software. With the associated Lagrange multiplier, the proposed distortion criterion can achieve 12.98% bitrate saving compared with SSD based RDO method on average.