A DDDAMS-based planning and control framework for surveillance and crowd control via UAVs and UGVs

•An effective and efficient control framework for UAV/UGV coordination is proposed.•Agent-based simulation is used for planning as well as real-time control purposes.•Filtering is used for UGV and UAV for tracking individuals and crowd, respectively.•Motions of UAV/UGV are determined based on minimal distance and energy consumption.•The framework is demonstrated for a hardware-in-the-loop simulation environment.

A dynamic data driven adaptive multi-scale simulation (DDDAMS) based planning and control framework is proposed for effective and efficient surveillance and crowd control via UAVs and UGVs. The framework is mainly composed of integrated planner, integrated controller, and decision module for DDDAMS. The integrated planner, which is designed in an agent-based simulation (ABS) environment, devises best control strategies for each function of (1) crowd detection (vision algorithm), (2) crowd tracking (filtering), and (3) UAV/UGV motion planning (graph search algorithm). The integrated controller then controls real UAVs/UGVs for surveillance tasks via (1) sensory data collection and processing, (2) control command generation based on strategies provided by the decision planner for crowd detection, tracking, and motion planning, and (3) control command transmission via radio to the real system. The decision module for DDDAMS enhances computational efficiency of the proposed framework via dynamic switching of fidelity of simulation and information gathering based on the proposed fidelity selection and assignment algorithms. In the experiment, the proposed framework (involving fast-running simulation as well as real-time simulation) is illustrated and demonstrated for a real system represented by hardware-in-the-loop (HIL) real-time simulation integrating real UAVs, simulated UGVs and crowd, and simulated environment (e.g. terrain). Finally, the preliminary results successfully demonstrate the benefit of the proposed dynamic fidelity switching concerning the crowd coverage percentage and computational resource usage (i.e. CPU usage) under cases with two different simulation fidelities.