Image partitioning and illumination in image-based pose detection for teleoperated flexible endoscopes

ObjectiveColorectal cancer is one of the leading causes of cancer-related deaths in the world, although it can be effectively treated if detected early. Teleoperated flexible endoscopes are an emerging technology to ease patient apprehension about the procedure, and subsequently increase compliance. Essential to teleoperation is robust feedback reflecting the change in pose (i.e., position and orientation) of the tip of the endoscope. The goal of this study is to first describe a novel image-based tracking system for teleoperated flexible endoscopes, and subsequently determine its viability in a clinical setting. The proposed approach leverages artificial neural networks (ANNs) to learn the mapping that links the optical flow between two sequential images to the change in the pose of the camera. Secondly, the study investigates for the first time how narrow band illumination (NBI) – today available in commercial gastrointestinal endoscopes – can be applied to enhance feature extraction, and quantify the effect of NBI and white light illumination (WLI), as well as their color information, on the strength of features extracted from the endoscopic camera stream.Methods and materialsIn order to provide the best features for the neural networks to learn the change in pose based on the image stream, we investigated two different imaging modalities – WLI and NBI – and we applied two different spatial partitions – lumen-centered and grid-based – to create descriptors used as input to the ANNs. An experiment was performed to compare the error of these four variations, measured in root mean square error (RMSE) from ground truth given by a robotic arm, to that of a commercial state-of-the-art magnetic tracker. The viability of this technique for a clinical setting was then tested using the four ANN variations, a magnetic tracker, and a commercial colonoscope. The trial was performed by an expert endoscopist (>2000 lifetime procedures) on a colonoscopy training model with porcine blood, and the RMSE of the ANN output was calculated with respect to the magnetic tracker readings. Using the image stream obtained from the commercial endoscope, the strength of features extracted was evaluated.ResultsIn the first experiment, the best ANNs resulted from grid-based partitioning under WLI (2.42 mm RMSE) for position, and from lumen-centered partitioning under NBI (1.69° RMSE) for rotation. By comparison, the performance of the tracker was 2.49 mm RMSE in position and 0.89° RMSE in rotation. The trial with the commercial endoscope indicated that lumen-centered partitioning was the best overall, while NBI outperformed WLI in terms of illumination modality. The performance of lumen-centered partitioning with NBI was 1.03 ± 0.8 mm RMSE in positional degrees of freedom (DOF), and 1.26 ± 0.98° RMSE in rotational DOF, while with WLI, the performance was 1.56 ± 1.15 mm RMSE in positional DOF and 2.45 ± 1.90° RMSE in rotational DOF. Finally, the features extracted under NBI were found to be twice as strong as those extracted under WLI, but no significance in feature strengths was observed between a grayscale version of the image, and the red, blue, and green color channels.ConclusionsThis work demonstrates that both WLI and NBI, combined with feature partitioning based on the anatomy of the colon, provide valid mechanisms for endoscopic camera pose estimation via image stream. Illumination provided by WLI and NBI produce ANNs with similar performance which are comparable to that of a state-of-the-art magnetic tracker. However, NBI produces features that are stronger than WLI, which enables more robust feature tracking, and better performance of the ANN in terms of accuracy. Thus, NBI with lumen-centered partitioning resulted the best approach among the different variations tested for vision-based pose estimation. The proposed approach takes advantage of components already available in commercial gastrointestinal endoscopes to provide accurate feedback about the motion of the tip of the endoscope. This solution may serve as an enabling technology for closed-loop control of teleoperated flexible endoscopes.