Using affective human–machine interface to increase the operation performance in virtual construction crane training system: A novel approach

In the construction industry, some progress have been achieved by researchers to design and implement environments for task training using VR technology and its derivatives such as Augmented and Mixed Reality. Although, these developments have been well recognized at the application level, however crucial to the virtual training system is the effective and reliable measurement of training performance of the particular skill and handling the experiment for long-run. It is known that motor skills cannot be measured directly, but only inferred by observing behaviour or performance measures. The typical way of measuring performance is through measuring task completion time and accuracy, but can be supported by indirect measurement of some other factors. In this paper, a virtual crane training system has been developed which can be controlled using control commands extracted from facial gestures and is capable to lift up loads/materials in the virtual construction sites. Then, we integrate affective computing concept into the conventional VR training platform for measuring the cognitive load and level of satisfaction during performance using human's forehead bioelectric-signals. By employing the affective measures and our novel control scheme, the designed interface could be adapted to user's affective status during the performance in real-time. This adaptable user interface approach helps the trainee to cope with the training for long-run performance, leads to gaining more expertise and provides more effective transfer of learning to other operation environments. The detailed methodology of the affective control is presented in the paper. The results and future applications of the proposed method for disabled users, especially from neck down are discussed.

Research Highlights
► In this study, affective measures from brain activities were employed for measuring the cognitive load and level of satisfaction during the task performance.
► These affective ( emotional) measures then were used to update the control unit of the human-machine interface.
► The affective and adaptable interface helps its users to cope with the training for long-run operation and increases the user's performance.