Don't just sign use brain too: A novel multimodal approach for user identification and verification

In this paper, we propose a novel multimodal user identification and verification scheme combining two inter-linked biometric traits, i.e., signature and brain signals (Electroencephalography (EEG)). Moulding handwritten strokes for signature stimulates EEG signals in the brain. The response of the brain signals during signing is unique for each user and this brain signal-signature relation is utilized in the proposed scheme. To the best of our knowledge, there exist no authentication approach combining these two inter-linked biometric traits. The proposed multimodal scheme uses Hidden Markov Model (HMM) based sequential classifier to model features extracted from signatures and EEG signals individually. Pyramid Histogram of Orientation Gradients (PHOG) features are extracted from the signature-images and next PHOG features are used to build user specific signature-HMM models. Similarly, user wise EEG-HMM models are built with Daubechies-4 (DB4) wavelet analysis. Finally, a score combining classification scores of signature-HMM and EEG-HMM models is used to perform user identification and verification. To evaluate the effectiveness of the proposed scheme, we have developed a dataset collecting these two traits simultaneously using the Emotiv Epoc+ device and pen-paper for 70 individual subjects. Thereafter, user's identification is performed with individuals' signature and EEG signals as well as their combined traits. The identification accuracy of the proposed multimodal approach has been achieved upto 98.24%. The effectiveness of the verification scheme is validated by the Receiver Operating Characteristic (ROC) curves.