Depth Sensor Based Automatic Hand Region Extraction by Using Time-series Curve and Its Application to Japanese Finger-spelled Sign Language Recognition

Hand sign recognition is one of most challenging issues in computer vision and human computer interaction, and many researchers tackle this issue. In this research, we focus on JFSL (Japanese Finger-spelled Sign Language) which is one of hand signs. The tasks for achieving high performance of JFSL recognition as well as other hand signs are how to extract hand region precisely and how to recognize hand signs accurately. To deal with the former task, in this paper, we propose an automatic hand region extraction method with a depth sensor. The characteristic points of our proposed method are to utilize Time-Series Curve, which is one of contour features, and to extract hand region accurately without wearing landmark object such as a color wristband. On the other hand, to tackle the latter task, in this research, we focus on a deep neural network based recognition method since such a method is reported that it allows us to achieve high performance for various recognition tasks. Therefore, in this paper, we investigate JFSL recognition performance with a deep neural network approach compared to that with the conventional image recognition method (HOG+SVM). From the experimental results with 8 subjects, we have confirmed that our proposed method allows us to extract hand region accurately regardless of subjects and JFSL signs. In addition, from the experimental results with a deep neural network based recognition method for JFSL recognition, we have achieved at least average recognition rate over 88%.