Situation-dependent remote AR collaborations: Image-based collaboration using a 3D perspective map and live video-based collaboration with a synchronized VR mode

The development of the Internet and smart devices has made it possible to support remote collaborations, which allows the expert and the worker to share information and co-work together in various and dispersed industrial fields. However, collaborations that exchange only audio and video information have limitations in the guided instruction of tasks, training of workers, and understanding of the situation. As augmented reality (AR) technology is considered to provide a more intuitive and immersive visualization and interaction in physical workspaces, many research works have been conducted to integrate AR into remote collaboration. Usually, there are two different types of AR collaboration depending on the shared media: 1) image-based AR collaboration and 2) live video-based AR collaboration. However, most of the previous research works cannot effectively support not only visual augmentation suitable for the situation but also accurate and effective annotations in the shared AR space. In this paper, we propose a situation-dependent remote AR collaboration approach that can selectively support either image- or live video-based AR collaborations: 1) image-based AR collaboration using a 3D perspective map and 2) live video-based AR collaboration with a synchronized VR mode. In particular, in the case of a certain situation with a limited network connection or limited HW/SW capabilities of the smart device, the image-based AR collaboration is more preferred. However, existing approaches cannot provide an integrated AR space from shared images. The proposed image-based AR collaboration enables to construct a 3D perspective map from the shared images taken in three or more directions, which can provide an integrated AR space for more effective AR annotations for remote collaboration. In the case of a normal situation without such problems, the live video-based AR collaboration should be supported. However, previous works have inherent problems such as inaccurate and mismatched AR annotations when the viewpoint of the live video is changed. The proposed live video-based AR collaboration with a synchronized VR mode can provide more effective and accurate 3D annotations by synchronizing virtual objects with physical objects. In particular, the VR mode can provide a complementing VR view of the AR-based physical space. In addition, through quantitative and qualitative experimental evaluation, we have conducted comparative studies with previous works. The results of this research show that the approach presented in this research has higher qualitative evaluation such as human behavior and task usability, as well as higher quantitative evaluation such as task performance and accuracy. Therefore, it is expected that the proposed approach can provide more user-oriented collaboration that considers user's situation and device performance in various industrial applications.