An inverse Yarbus process: Predicting observers' task from eye movement patterns

•A computational model is developed for visual-task inference given eye trajectories.•Hidden Markov Models are used to predict fixation coordinates contingent on task.•The model allows for both overt and covert shifts of attention.•The model locates attended targets and identifies the task.•The results support the findings of Yarbus on task-dependent eye movements.

In this paper we develop a probabilistic method to infer the visual-task of a viewer given measured eye movement trajectories. This method is based on the theory of hidden Markov models (HMM) that employs a first order Markov process to predict the coordinates of fixations given the task. The prediction confidence level of each task-dependent model is used in a Bayesian inference formulation, whereby the task with the maximum a posteriori (MAP) probability is selected. We applied this technique to a challenging dataset consisting of eye movement trajectories obtained from subjects viewing monochrome images of real scenes tasked with answering questions regarding the scenes. The results show that the HMM approach, combined with a clustering technique, can be a reliable way to infer visual-task from eye movements data.