Multi-modal user interaction method based on gaze tracking and gesture recognition

This paper presents a gaze tracking technology which provides a convenient human–centric interface for multimedia consumption without any wearable device. It enables a user to interact with various multimedia on a large display in distance by tracking user movement and acquiring high resolution eye images. This paper also presents a gesture recognition technology which is helpful to interact with scene descriptions in terms of controlling and rendering scene objects. It is based on Hidden Markov Model and CRF using a commercial depth sensor. And then, this paper shows a collaboration method with those new sensors and MPEG standards in order to achieve interoperability among interactive applications, new user interaction devices and users.

► Gaze tracker enables a user to interact with various multimedia on a large display from a distance. ► Gaze tracker uses an IR device and its reflectance with calibrated wide-angle and narrow-angle cameras. ► Gesture recognizer proposes a novel majority voting method for accumulated gesture feature sequences. ► We show a collaboration method with gaze tracker, gesture recognizer, and MPEG standards.