Action recognition from only somatosensory information using spectral learning in a hidden Markov model

• This paper proposes an approach to classifying somatosensory information in the human full body into the action categories.
• The muscle activities in the human full body are estimated from captured motions, ground reaction forces, and EMG data.
• The discrete hidden Markov models to be optimized by spectral learning are adopted for the action classifiers.

Human action classification is fundamental technology for robots that have to interpret a human’s intended actions and make appropriate responses, as they will have to do if they are to be integrated into our daily lives. Improved measurement of human motion, using an optical motion capture system or a depth sensor, allows robots to recognize human actions from superficial motion data, such as camera images containing human actions or positions of human bodies. But existing technology for motion recognition does not handle the contact force that always exists between the human and the environment that the human is acting upon. More specifically, humans perform feasible actions by controlling not only their posture but also the contact forces. Furthermore these contact forces require appropriate muscle tensions in the full body. These muscle tensions or activities are expected to be useful for robots observing human actions to estimate the human’s somatosensory states and consequently understand the intended action. This paper proposes a novel approach to classifying human actions using only the activities of all the muscles in the human body. Continuous spatio-temporal data of the activity of an individual muscle is encoded into a discrete hidden Markov model (HMM), and the set of HMMs for all the muscles forms a classifier for the specific action. Our classifiers were tested on muscle activities estimated from captured human motions, electromyography data, and reaction forces. The results demonstrate their superiority over commonly used HMM-based classifiers.