Multimodal approach for polysensory stimulation and diagnosis of subjects with severe communication disorders

An experimental multimodal system, designed for polysensory diagnosis and stimulation of non-communicative subjects, with severe brain injuries is presented. The user interface uses an eye-tracking device and EEG monitoring of the subject. The system is evaluated on 9 patients, data analysis methods are described, and experiments of correlating Glasgow Coma Scale with extracted features describing subjects performance in therapeutic exercises exploiting EEG and eyetracker are presented. Performance metrics are proposed, and k-means clusters used to define concepts for mental states related to EEG and eyetracking activity. Finally, it is shown that the strongest correlations are between the number of detected mental states and GCSe score, and between maximal length of mental state and GCSm. Weaker correlations are reported as well. Moreover an approach to classification of real and imaginary motion of limbs is presented and discussed. Classifiers based on SVM, Artificial Neural Networks, and Rough Sets were trained and accuracy reaching 91% for the real, and up to 100% for the imaginary type of motion was observed. Assessments of communication skills and therapy is possible with the system, already employed in long-term care facility.