Using a game controller for relaying deictic gestures in computer-mediated communication

In computer-mediated communication, participants often experience a loss of information that would have been conveyed by non-verbal means in a face-to-face communication. They need to compensate for this loss by being more verbose which may have a negative impact on the efficiency of communication or give rise to misunderstandings. In this paper, we present a computer system that augments the imagery perceived from the remote site with additional visual information and that can be controlled by using a Nintendo Wiimote game controller. Our prototype illustrates our approach how a system might be able to partially compensate for the information typically lost in conventional video conferencing where deictic gestures cannot be relayed properly to the remote site. Concerning user acceptance, our user tests also show that the visualization technique is crucial which is used for highlighting the objects or persons that are being pointed at. Adequate visualization techniques based on non-photorealistic rendering are proposed.

Research highlights► Game controllers are usable to relay deictic gestures in video-conferencing. ► Imagery from the remote site has to be augmented with visual information. ► Visualization techniques used are crucial: highlighted areas should not be modified. ► User tests show how participants in a video-conference embrace the novel features.