Inferring action structure and causal relationships in continuous sequences of human action

• We present a Bayesian rational analysis of action segmentation using statistical and causal cues.
• Like our model, people use statistical regularities and causal structure to segment fluid action.
• People can identify the relevant causal variables from within a continuous sequence of events.
• Similarities in word and action segmentation suggest a more general statistical learning ability.

In the real world, causal variables do not come pre-identified or occur in isolation, but instead are embedded within a continuous temporal stream of events. A challenge faced by both human learners and machine learning algorithms is identifying subsequences that correspond to the appropriate variables for causal inference. A specific instance of this problem is action segmentation: dividing a sequence of observed behavior into meaningful actions, and determining which of those actions lead to effects in the world. Here we present a Bayesian analysis of how statistical and causal cues to segmentation should optimally be combined, as well as four experiments investigating human action segmentation and causal inference. We find that both people and our model are sensitive to statistical regularities and causal structure in continuous action, and are able to combine these sources of information in order to correctly infer both causal relationships and segmentation boundaries.