Complementing visual tracking of moving targets by fusion of tactile sensing

Robot control in uncertain and dynamic environments can be greatly improved using sensor-based control. Vision is a versatile low-cost sensory modality, but low sample rate, high sensor delay and uncertain measurements limit its usability, especially in strongly dynamic environments. Vision can be used to estimate a 6-DOF pose of an object by model-based pose-estimation methods, but the estimate is typically not accurate along all degrees of freedom. Force is a complementary sensory modality allowing accurate measurements of local object shape when a tooltip is in contact with the object. In multimodal sensor fusion, several sensors measuring different modalities are combined together to give a more accurate estimate of the environment. As force and vision are fundamentally different sensory modalities not sharing a common representation, combining the information from these sensors is not straightforward. We show that the fusion of tactile and visual measurements enables to estimate the pose of a moving target at high rate and accuracy. Making assumptions of the object shape and carefully modeling the uncertainties of the sensors, the measurements can be fused together in an extended Kalman filter. Experimental results show greatly improved pose estimates with the proposed sensor fusion.