Design and evaluation of freehand menu selection interfaces using tilt and pinch gestures

Freehand gestural interaction, in which the user’s hands move in mid-air to provide input, has been of interest to researchers, but freehand menu selection interfaces have been under-investigated so far. Freehand menu selection is inherently difficult, especially with increasing menu breadth (i.e., the number of items), largely because moving hands in free space cannot achieve precision as high as physical input devices such as mouse and stylus. We have designed a novel menu selection interface called the rapMenu (Ni et al., 2008), which is controlled by wrist tilt and multiple pinch gestures, and takes advantage of the multiple discrete gesture inputs to reduce the required precision of the user hand movements.In this article, we first review the visual design and behavior of the rapMenu technique, as well as related design issues and its potential advantages. In the second part, we present two studies of the rapMenu in order to further investigate the strengths and limitations of the design principle. In the first study, we compared the rapMenu to the extensively studied tilt menu technique (Rahman et al., 2009). Our results revealed that the rapMenu outperforms the tilt menu as menu breadth increases. In the second study, we investigated how the rapMenu affords the opportunity of eyes-free selection and users’ transition from novice to expert. We found that within 10 min of practice, eyes-free selection with rapMenu has competitive speed and accuracy with the visual rapMenu and the tilt menu. Finally, we discuss design variations that use other axes of wrist movement and adopt alternative auditory feedback.

► We evaluate the rapMenu, a menu interface controlled by freehand tilt and pinch gestures.
► Multiple discrete gestures are used to reduce the required precision of continuous gestures.
► With increasing menu breadth, the rapMenu outperforms the menu that is controlled solely by tilt.
► The rapMenu allows for eyes-free selection and users’ transition from novice to expert.