Gesture recognition: A review focusing on sign language in a mobile context

Sign languages, which consist of a combination of hand movements and facial expressions, are used by deaf persons around the world to communicate. However, hearing persons rarely know sign languages, creating barriers to inclusion. The increasing progress of mobile technology, along with new forms of user interaction, opens up possibilities for overcoming such barriers, particularly through the use of gesture recognition through smartphones. This Literature Review discusses works from 2009 to 2017 that present solutions for gesture recognition in a mobile context as well as facial recognition in sign languages. Among a diversity of hardware and techniques, sensor-based gloves were the most used special hardware, along with brute force comparison to classify gestures. Works that did not adopt special hardware mostly used skin color for feature extraction in gesture recognition. Classification algorithms included: Support Vector Machines, Hierarchical Temporal Memory and Feedforward backpropagation neural network, among others. Recognition of static gestures typically achieved results higher than 80%. Fewer papers recognized dynamic gestures, obtaining results above 90%. However, most experiments were performed under controlled environments, with specific lighting conditions, and were only using a small set of gestures. In addition, the majority of works dealt with a simple background and used special hardware (which is often cumbersome for the user) to facilitate feature extraction. Facial expression recognition achieved high classification results using Random-Forest and Multi-layer Perceptron. Despite the progress being made with the increasing interest in gesture recognition, there are still important gaps to be addressed in the context of sign languages. Besides improving usability and efficacy of the solutions, recognition of facial expression and of both static and dynamic gestures in complex backgrounds must be considered.