MARIO: Mid-air Augmented Reality Interaction with Objects

•Users can interact with mid-air images with physical objects or their hands.•Users can see the mid-air images without special glasses.•The mid-air images can move in a 3D space measuring 350 (W) × 300 (D) × 250 (H) mm.•Shadows are cast by projected images to give users a sense of reality.•User study has proven the popularity of our system as an entertainment application.

A seamless connection between a game space and the real world is a long-sought goal of the entertainment computing field. Recent games have intuitive interactions through gestures and motion-based control. However, their images are still confined to the displays. In this paper, we propose a novel interaction experience in which mid-air images interact with physical objects. Our “Mid-air Augmented Reality Interaction with Objects” (MARIO) system enables images to be displayed in 3D spaces beyond screens. By creating a spatial link between images appearing in mid-air and physical objects, we extend video games into the real world. For entertainment purposes, a game character appears in mid-air and jumps over real blocks that users have arranged with their hands. A user study was conducted at public exhibitions, and the results showed the popularity and effectiveness of the system as an entertainment application.