Development of a rescue system for agricultural machinery operators using machine vision

In this study, an automatic rescue system was proposed to monitor agricultural machinery operators using machine vision. The rescue system was developed to recognise the driver inattention status, that is, the distraction and fatigue by recognising the driver's actions. A Kinect sensor was used to collect image sequences of the operators, and the recognition system relied on the “player extraction” function of the Kinect sensor. A Hankel-based Kernel Mutual Subspace Method (KMSM) was developed to monitor tractor drivers and recognise driver inattention behaviours. To reduce the computational complexity for fulfilling the requirements of recognition, low-dimensional image vectors were used to generate low-dimensional block Hankel matrixes as representations for input action sequences. To evaluate the performance of the proposed KMSM, a driver action dataset was established that included 10 tractor drivers and 5 types of action that denote inattention. The drivers' inattention actions were classified into three danger levels, and the corresponding countermeasures for the actions at each danger level were similarly classified. Both offline and online experiments using similar subjects and different subjects were conducted to evaluate the designed inattention action recognition algorithm. In the offline experiment, the proposed Hankel-based KMSM achieved recognition rates of 91.18% and 86.18% when using similar and different subjects, respectively; and in the online experiment, the proposed method achieved 87.02 and 79.97% when using similar and different subjects, respectively. The average computation time of the Hankel-based KMSM was 0.07 s in the online experiment. Thus, the proposed Hankel-based KMSM method satisfies both the accuracy and the real-time requirements for a driver rescue system.