A nearest neighbor approach for fruit recognition in RGB-D images based on detection of convex surfaces

Automatic fruit picking is a challenging problem in robotics with a wide application field. A prerequisite for realization of a robotic fruit picker is its ability to detect fruits in tree tops. An expert system, which would be able to compete with human perception, must be capable of recognizing fruits among leaves and branches under uncontrolled conditions, where fruits are occluded and shaded. In this paper, a novel approach for fruit recognition in RGB-D images based on detection and classification of convex surfaces is proposed. The input RGB-D image is first segmented into convex surfaces by a region growing procedure. Each convex surface is then described by an appropriate descriptor and classified with the aid of the associated descriptor. A novel descriptor of approximately convex surfaces is proposed, which we named Convex Template Instance (CTI) descriptor. It is based on approximating surfaces by convex polyhedrons with quantized face orientations, where every polyhedron face corresponds to one descriptor component. Computation of the proposed descriptor is simple and can be performed very efficiently. The proposed CTI descriptor is compared to the SHOT descriptor, a standard descriptor for 3D point clouds. Two variants of the both CTI and SHOT descriptor are evaluated, a variant which uses color and a variant which does not. A k-nearest neighbor classifier is used to classify detected surfaces into two classes: fruit and other. The main advantage of the proposed expert system in comparison to other fruit recognition solutions is its computational efficiency, which is of great importance for its target application - an automatic fruit picker. The proposed approach is evaluated using a challenging dataset containing RGB-D images of four fruit sorts acquired under uncontrolled conditions, which has been made publicly available to the scientific community, allowing benchmarking of novel fruit recognition methods.