Continuous detection of human fall using multimodal features from Kinect sensors in scalable environment

•A reliable method for fall detection by combining skeleton and RGB from Kinect sensor is proposed.•An online fall detection is efficiently solved thanks to the fast computation of skeleton or fall candidate detection based on grayscale motion map.•A client-server architecture within a Kinect network is developed that is easy to scale for any space size.

Background and Objectives: Automatic detection of human fall is a key problem in video surveillance and home monitoring. Existing methods using unimodal data (RGB / depth / skeleton) may suffer from the drawbacks of inadequate lighting condition or unreliability. Besides, most of proposed methods are constrained to a small space with off-line video stream.Methods: In this study, we overcome these encountered issues by combining multi-modal features (skeleton and RGB) from Kinect sensor to take benefits of each data characteristic. If a skeleton is available, we propose a rules based technique on the vertical velocity and the height to floor plane of the human center. Otherwise, we compute a motion map from a continuous gray-scale image sequence, represent it by an improved kernel descriptor then input to a linear Support Vector Machine. This combination speeds up the proposed system and avoid missing detection at an unmeasurable range of the Kinect sensor. We then deploy this method with multiple Kinects to deal with large environments based on client server architecture with late fusion techniques.Results: We evaluated the method on some freely available datasets for fall detection. Compared to recent methods, our method has a lower false alarm rate while keeping the highest accuracy. We also validated on-line our system using multiple Kinects in a large lab-based environment. Our method obtained an accuracy of 91.5% at average frame-rate of 10fps.Conclusions: The proposed method using multi-modal features obtained higher results than using unimodal features. Its on-line deployment on multiple Kinects shows the potential to be applied in to any of living space in reality.