Detection of passion fruits and maturity classification using Red-Green-Blue Depth images

A machine vision algorithm was developed to detect passion fruits and identify maturity of the detected fruits using natural outdoor RGB-D images. As different passion fruits on the same branch can be in different maturity stages, detection and maturity classification on a complex background are very important for yield mapping and development of intelligent mobile fruit-picking robots. In this study, a Kinect sensor was used for data acquisition, and maturity stages of the fruits were divided into five categories: young (Y), near-young (NY), near-mature (NM), mature (M) and after-mature (AM). The algorithm involved two stages. First, by colour and depth images, passion fruits were detected using faster region-based convolutional neural networks (Faster R-CNN), and colour-based detection was integrated with depth-based detection for improving detection performance. Second, for each detected fruit region, the dense scale invariant features transform (DSIFT) algorithm combined with locality-constrained linear coding (LLC) was used to extract and represent the features of fruit maturity from R, G, and B channels, respectively. In addition, the RGB-DSIFT-LLC features were input into a linear support vector machine (SVM) classifier for identifying the maturity of fruits. By conducting an experimental study on a special dataset, we verified that the proposed method achieves 92.71% detection accuracy and 91.52% maturity classification accuracy.