Multi-spectral and multi-perspective video arrays for driver body tracking and activity analysis

This paper presents a novel approach to recognizing driver activities using a multi-perspective (i.e., four camera views) multi-modal (i.e., thermal infrared and color) video-based system for robust and real-time tracking of important body parts. The multi-perspective characteristics of the system provides redundant trajectories of the body parts, while the multi-modal characteristics of the system provides robustness and reliability of feature detection and tracking. The combination of a deterministic activity grammar (called ‘operation-triplet’) and a Hidden Markov model-based classifier provides semantic-level analysis of human activity. The application context for this research is that of intelligent vehicles and driver assistance systems. Experimental results in real-world street driving demonstrate effectiveness of the proposed system.