Autostereoscopic augmented reality visualization for depth perception in endoscopic surgery

Augmented reality (AR) has received increasing attention in minimally invasive surgery (MIS) applications. The goal of applying AR techniques to MIS is to enhance a surgeon's perception of the spatial relationship by overlaying invisible structures (e.g. tumor or vessels) onto the in vivo endoscopic video acquired during the surgery. One of primary issues of AR visualization is to provide correct depth perception for visible and invisible structures. In this paper, we present a video-based AR system consisting of functional modules for real-time 3D surface capture, reconstruction, and registration with pre-operative segmented CT model. The real-time 3D registration allows precise overlay of invisible structures onto 2D video for AR visualization. The AR overlay result is displayed on a multi-view autostereoscopic lenticular LCD. To study and compare the efficacy of AR visualization techniques, we investigated five different AR visualization modes. Both simulated and in vivo experiments were carried out and autostereoscopic AR visualization results were given. Evaluation and comparison for depth perception between five AR visualization modes are presented. Finally, we conclude the characteristics of these visualization modes. The novelty of our work lies in successful implementation of an end-to-end 3D autostereoscopic AR system from real-time reconstruction and registration with our multi-channel 3D endoscope, and systematic evaluation and comparison of five different visualization modes for depth perception.