A Virtual World Application to Assess and Aid Human Machine Collaboration

This paper presents an application using virtual reality (VR) environments to create serious games. A brief introduction to the problem area is presented followed by a detailed discussion on use of VR technology to aid research related to understanding human machine team collaborations. Some of the technical challenges and ways to mitigate them are also discussed. A VR platform provides a rich environment to setup complex scenarios that includes an avatar-based representation of human machine teams. Integration of physiological sensors into the VR application enables the computer to interpret the human state and modulate its behavior accordingly. Multimodal interactions between humans and machines allow for richer communication between them. In the current research, serious games are designed to facilitate human and computer avatar teams working together to accomplish a goal. An important aspect of this research is to study such a teaming in a high stress environment where the mission is time-critical and of high risk. The computer avatar has a social component that is driven by the RULER model (developed at Emotional Intelligence lab at Yale University) to help R: Recognize emotions in self and others, U: Understand emotion's causes and consequences, L: Label emotions with an accurate and diverse vocabulary, E: Express emotions in socially appropriate ways, and R: Regulate emotions in socially appropriate ways (Ruler Group, 2013).We plan to study the change in human computer relationship as time progresses and complexity of the task increases. A successful test of the social skills module would show improved interactions in the virtual world measured by such variables as being influenced by VR recommendations, liking the computer teammate better, and performing better on the VR tasks.