On surveillance for safety critical events: In-vehicle video networks for predictive driver assistance systems

•A distributed camera-sensor system for driver assistance and situational awareness.•Systematic, comparative evaluation of cues for prediction of safety critical events.•Real-time prediction of overtaking and braking maneuvers.•Detailed temporal analysis of the utility of various cues for maneuver prediction.•Early prediction (1–2 s) before the maneuver is shown on real-world data.

We study techniques for monitoring and understanding real-world human activities, in particular of drivers, from distributed vision sensors. Real-time and early prediction of maneuvers is emphasized, specifically overtake and brake events. Study this particular domain is motivated by the fact that early knowledge of driver behavior, in concert with the dynamics of the vehicle and surrounding agents, can help to recognize dangerous situations. Furthermore, it can assist in developing effective warning and driver assistance systems. Multiple perspectives and modalities are captured and fused in order to achieve a comprehensive representation of the scene. Temporal activities are learned from a multi-camera head pose estimation module, hand and foot tracking, ego-vehicle parameters, lane and road geometry analysis, and surround vehicle trajectories. The system is evaluated on a challenging dataset of naturalistic driving in real-world settings.