Handling occlusions in augmented reality based on 3D reconstruction method

The correct relationships between real and virtual objects are of utmost importance to a realistic augmented reality system, in which the occlusion handling method should be able to estimate the spatial relationships between real and virtual objects, as well as handle the mutual occlusion automatically in real-time. To accomplish the above tasks simultaneously, we propose a novel occlusion handling method based on 3D reconstruction, which consists of offline stage and online stage. In the offline stage, we get the depth map of the real scene using a low cost RGB-D camera. Then the 3D coordinate of each point in the global coordinate system are obtained and will be used in the online occlusion handling stage. In the online stage, we design a GPU based 3D point clouds alignment method by using point to tangent plane distance as error metric to accelerate the convergence speed and reduce the iterations. The correct relationships between real and virtual objects are then obtained automatically by comparing each pixel’s Z coordinate value of real objects with that of virtual objects in a smaller region to achieve real-time performance. More specifically, we can judge and handle the mutual occlusion without human interactivity in real time, and experimental results prove its effectiveness.