Temporal multimodal data synchronisation for the analysis of a game driving task using EEG

• A novel and generic method for precise temporal synchronisation of multimodal data streams is presented.
• Tight temporal synchronisation with commodity data logging tools can be achieved.
• Driving patterns are correlated to brain activities.
• Synchronised multimodal logging enabled documenting the process of learning a driving task.

Multimodal data channels such as bio-physiological signals are increasingly used in game-play studies to better understand players’ behaviours and their motivations. It is however difficult to perform any sort of conclusive analysis solely based on bio-physiological signals due to the complex nature of epistemic, semiotic and ergotic activities surrounding in-game activities and the artefacts facilitating player immersion. Thus a combined analysis of multiple data streams including in-game data and bio-physiological signals is indispensable to produce contextualised information from which a deep analysis of game mechanics and their effects can be performed.Precise synchronisation in capturing multiple streams is required to generate valid inter-stream correlations and meaningful information. Typically there are no automatic mechanisms built in the game architecture or in commercial data logging systems for multimodal data synchronisation and data fusion. This paper presents a novel and generic technique based on inducing identifiable signature pulses in data channels to accurately synchronise multiple temporal data streams. This technique is applied and its capabilities are exhibited using a driving game simulation as an exemplar. In this example, driver’s in-game behavioural data is synchronised and correlated with their temporal brain activity. The concept of simplex method borrowed from linear programming is used to correlate between the driving patterns and brain activity in this initial study is provided so as to allow studying/investigating user behaviour in relation to learning of the driving track.