Augmenting video surveillance footage with virtual agents for incremental event evaluation

The fields of segmentation, tracking and behavior analysis demand for challenging video resources to test, in a scalable manner, complex scenarios like crowded environments or scenes with high semantics. Nevertheless, existing public databases cannot scale the presence of appearing agents, which would be useful to study long-term occlusions and crowds. Moreover, creating these resources is expensive and often too particularized to specific needs. We propose an augmented reality framework to increase the complexity of image sequences in terms of occlusions and crowds, in a scalable and controllable manner. Existing datasets can be increased with augmented sequences containing virtual agents. Such sequences are automatically annotated, thus facilitating evaluation in terms of segmentation, tracking, and behavior recognition. In order to easily specify the desired contents, we propose a natural language interface to convert input sentences into virtual agent behaviors. Experimental tests and validation in indoor, street, and soccer environments are provided to show the feasibility of the proposed approach in terms of robustness, scalability, and semantics.

Research highlights► We propose an augmented reality framework to increase the complexity of image sequences in terms of occlusions and crowds, in a scalable and controllable manner. ► Existing datasets can be increased with augmented sequences containing virtual agents. ► Such sequences are automatically annotated, thus facilitating evaluation in terms of segmentation, tracking, and behavior recognition. ► To easily specify the desired contents, we propose a natural language interface to convert input sentences into virtual agent behaviors. As a result, complex scenes like crowded environments or scenes with controllable behavior contents can be generated controllably and repeatedly from original scenes by incorporating virtual agents, making it easier to develop testing datasets for segmentation, tracking, and behavior analysis.