Low cost remote gaze gesture recognition in real time

Predefined sequences of eye movements, or ‘gaze gestures’, can be consciously performed by humans and monitored non-invasively using remote video oculography. Gaze gestures hold great potential in human–computer interaction, HCI, as long as they can be easily assimilated by potential users, monitored using low cost gaze tracking equipment and machine learning algorithms are able to distinguish the spatio-temporal structure of intentional gaze gestures from typical gaze activity performed during standard HCI. In this work, an evaluation of the performance of a bioinspired Bayesian pattern recognition algorithm known as Hierarchical Temporal Memory (HTM) on the real time recognition of gaze gestures is carried out through a user study. To improve the performance of traditional HTM during real time recognition, an extension of the algorithm is proposed in order to adapt HTM to the temporal structure of gaze gestures. The extension consists of an additional top node in the HTM topology that stores and compares sequences of input data by sequence alignment using dynamic programming. The spatio-temporal codification of a gesture in a sequence serves the purpose of handling the temporal evolution of gaze gestures instances. The extended HTM allows for reliable discrimination of intentional gaze gestures from otherwise standard human–machine gaze interaction reaching up to 98% recognition accuracy for a data set of 10 categories of gaze gestures, acceptable completion speeds and a low rate of false positives during standard gaze–computer interaction. These positive results despite the low cost hardware employed supports the notion of using gaze gestures as a new HCI paradigm for the fields of accessibility and interaction with smartphones, tablets, projected displays and traditional desktop computers.

Figure optionsDownload as PowerPoint slideHighlights
► This work presents an extension of the traditional HTM paradigm in order to improve HTM's performance in the real time recognition of saccadic gaze gestures.
► The extended HTM adapts HTM algorithms to problems were the spatio-temporal structure of instances unfolds over time.
► Our gaze gesture recognition system is able to robustly recognize gaze gestures in real time.
► The purpose of such system is to provide a communication channel between humans and computers, targeting specifically either users with disabilities or environments where traditional input channels are not suitable or could be augmented.
► The proposed extended HTM approach is not specifically designed to deal with gaze gestures specifically, since it is highly independent of hardware and preprocessing of input data.