Multimodal information improves the rapid detection of mental fatigue

One of the major challenge in the detection of mental states is improving the accuracy of brain activity-based detectors with additional information from extracranial signals. We assessed the suitability, for real-time mental fatigue detection, of EEG, EOG and ECG measurements, taken separately or together. Thirteen subjects performed six blocks of switching tasks. For each participant, the block with the lowest error rate from the first two blocks and the block with the highest error rate from the last three blocks were discriminated with a machine learning algorithm (support vector machine). The classification scores obtained with ECG or EOG were greater than would be expected by chance (>50%) for time windows of at least 8 s. EEG was the best single mode of detection, with classification scores ranging from 80 ± 3% with a 4 s time window to 94 ± 2% with a 30 s time window. The addition of ECG and EOG features to EEG features significantly increased classification scores for short time windows (e.g., to 86 ± 3% with a 4 s time window, p < 0.001). For short time windows (up to 12 s), ECG significantly increased the discriminatory power of EEG, whereas EOG did not. These results demonstrate that mental state detection on the basis of extracerebral measurements is feasible and that a combination of EEG and ECG is particularly appropriate for the rapid detection of mental fatigue.

► We classify EEG, EOG and ECG epochs of mental fatigue versus no fatigue.
► EEG is the best single modality.
► We combined the modalities using a multivariate classification algorithm.
► The combined ECG and EEG perform better than EEG alone for short time windows.
► ECG brings complementary information to that present in the EEG.