Multi-agent event recognition by preservation of spatiotemporal relationships between probabilistic models

•Modeling/recognition of multi-agent activities (American football plays).•Activities modeled as graphs, inexact graph matching used for comparison.•Single-agent activity represented as motion patterns, modeled as graph nodes.•Spatio-temporal relationships between single-agent behaviors modeled as graph edges.•We present our own dataset of football plays called “UCF Football”.

We present a new method for multi-agent activity analysis and recognition that uses low level motion features and exploits the inherent structure and recurrence of motion present in multi-agent activity scenarios.Our representation is inspired by the need to circumvent the difficult problem of tracking in multi-agent scenarios and the observation that for many visual multi-agent recognition tasks, the spatiotemporal description of events irrespective of agent identity is sufficient for activity classification.We begin by learning generative models describing motion induced by individual actors or groups, which are considered to be agents. These models are Gaussian mixture distributions learned by linking clusters of optical flow to obtain contiguous regions of locally coherent motion. These possibly overlapping regions or segments, known as motion patterns are then used to analyze a scene by estimating their spatial and temporal relationships. The geometric transformations between two patterns are obtained by iteratively warping one pattern onto another, whereas the temporal relationships are obtained from their relative times of occurrence within videos. These motion segments and their spatio-temporal relationships are represented as a graph, where the nodes are the statistical distributions, and the edges have geometric transformations between motion patterns transformed to Lie space, as their attributes. Two activity instances are then compared by estimating the cost of attributed inexact graph matching. We demonstrate the application of our framework in the analysis of American football plays, a typical multi-agent activity. The performance analysis of our method shows that it is feasible and easily generalizable.

Graphical abstractFigure optionsDownload full-size imageDownload high-quality image (270 K)Download as PowerPoint slide