Needle in a haystack: Interactive surgical instrument recognition through perception and manipulation

This paper presents a solution to the challenge of accurate surgical instrument recognition by a Robotic Scrub Nurse (RSN) in the Operating Room (OR). Surgical instruments, placed on the surgical mayo tray, are often cluttered, occluded and display specular light which poses a challenge for conventional recognition algorithms. To tackle this problem we resort to a hybrid computer vision and robotic manipulation combined strategy. The instruments are first segmented and pose is estimated, then the RSN system picks up the unknown instruments and presents them to the optical sensor in the determined pose. Last, the instruments are recognized and delivered. Experiments were conducted to evaluate the performance of the proposed segmentation, grasping and recognition algorithms, respectively. The proposed patch-based segmentation algorithm can achieve an F-score of 0.90. The proposed force-based grasping protocol can achieve an average picking success rate of 92% with various instrument layouts, and the proposed attention-based instrument recognition module can reach a recognition accuracy of 95.6%. Experimental results indicate the applicability and effectiveness of a RSN to perform accurate and robust surgical instrument recognition.