Edge-aware depth image filtering using color segmentation

•New method to improve edge quality of depth maps in video-plus-depth sequences.•New assessment method (EAT) to quantify edge enhancement ability of a depth filter.•Public database of video-plus-depth sequences to compute EAT scores.

High-quality depth maps are essential to many tasks in 3D vision. Especially depth maps generated with one of the recently upcoming low-cost 3D scanners need postprocessing to enhance depth quality. In this paper, we focus on one special aspect of depth quality: The alignment of color and depth edges in video-plus-depth sequences. Even with calibrated depth sensors, depth edges usually do not fully align with the edges of the color stream. We present a new depth enhancement method based on color segmentation that aligns depth edges with color edges. We also introduce a new assessment method called Edge Accuracy Test (EAT) to quantify the ability of a depth enhancement algorithm to care for aligned edges. We publish the video-plus-depth sequences of EAT so that other researchers can perform EAT on their algorithms.