Real-time sensory substitution to enable players who are blind to play video games using whole body gestures

• Real-time video analysis augments video games with additional vibrations.
• With this enhancement people who are blind can play standard video games.
• No significant difference in performance occurred between blind and sighted players.

Gesture-based interaction adds a new level of immersion to video games, but players who are blind are unable to play them as these games use visual cues to indicate what gesture to provide and when. Though visual cues can be substituted with audio or haptic cues, this often requires access to the source code, which is not attainable for commercial games. We present a solution that uses real-time video analysis to detect the presence of a particular visual cue, which is then substituted with a vibrotactile cue that is provided with an external controller. A user study with 28 sighted participants with a popular commercial gesture based game showed no significant difference in performance between visual and vibrotactile feedback. A follow up study with seven visually impaired participants revealed no significant difference in performance between both groups of users. Both studies demonstrate the feasibility of real-time sensory substitution as a cost-effective approach for making gesture-based video games accessible to players who are blind.