Calibration and low-level data fusion algorithms for a parallel 2D/3D-camera

In this article we propose a calibration algorithm and three low-level data fusion algorithms for a parallel 2D/3D-camera system. A parallel 2D/3D-camera is a hardware setup of a range camera and a high-resolution gray-value camera spatially related to each other by a fixed translation. The proposed calibration algorithm utilizes the fact that for known calibration patterns the range reconstruction accuracy of the gray-value camera is significantly higher than that of the range sensor. Using the calibrated 2D/3D-camera we identify the range pixels within the gray-value image for each pair of acquired camera images. We present three low-level data fusion approaches assigning range information to each gray-value pixel based on different neighborhood relations to the identified range pixels: one-nearest neighbor, nearest neighbors in the surrounding Delaunay-triangle, and nearest neighbors constrained by a gray-value image segmentation. We demonstrate the applicability, efficiency and accuracy of our calibration and fusion algorithms on real and synthetic data. Our real experiments are performed on a 2D/3D-camera comprising a Siemens 64 × 8-pixel time-of-flight range camera developed within the European project PReVENT (UseRCams) and a common gray-value camera.