A framework of mining semantic-based probabilistic event relations for complex activity recognition

Human activity recognition has become a key research topic in a variety of applications. Modeling activity events and their rich relations using high-level human understandable activity models such as semantic-based knowledge base hold promise. However, formulas in current semantic-based approaches are generally manually encoded, which is rather unrealistic in situations where event relations are intricate. Moreover, current approaches for learning event relations often lack the capability to handle uncertainties. To address these issues, we present a framework to learn an event knowledge base (EKB) of probabilistic interval-based event relations and use them to infer varied semantic-level queries about activity occurrences under uncertainty. Specifically, we formalize an activity model to represent eight temporal and hierarchical event relations and four commonly performed queries. We leverage pattern mining techniques to learn an EKB associated with these relations and queries in a unified way. Experimental results show that the proposed framework with the learned EKB involving temporal and hierarchical dependencies leads to a significant performance improvement on activity recognition, particularly in the presence of incomplete or incorrect observations.