A learning framework for semantic reach-to-grasp tasks integrating machine learning and optimization

The ability to implement semantic Reach-to-grasp (RTG) tasks successfully is a crucial skill for robots. Given unknown objects in an unstructured environment, finding an feasible grasp configuration and generating a constraint-satisfied trajectory to reach it are challenging. In this paper, a learning framework which combines semantic grasp planning with trajectory generation is presented to implement semantic RTG tasks. Firstly, the object of interest is detected by using an object detection model trained by deep learning. A Bayesian-based search algorithm is proposed to find the grasp configuration with highest probability of success from the segmented image of the object using a trained quality network. Secondly, for robotic reaching movements, a model-based trajectory generation method inspired by the human internal model theory is designed to generate a constraint-satisfied trajectory. Finally, the presented framework is validated both in comparative analysis and on real-world experiments. Experimental results demonstrated that the proposed learning framework enables the robots to implement semantic RTG tasks in unstructured environments.