A perceptually based spatio-temporal computational framework for visual saliency estimation

•Spatio-temporal computational framework based on psychological evidences.•Spatio-temporal and static energies by using the same multiscale 3D Gabor filterbank.•Motion information in different scales for both luminance and color stream modalities.•A new movie database with eye-tracking annotation.•Significant improvements on spatio-temporal saliency estimation.

The purpose of this paper is to demonstrate a perceptually based spatio-temporal computational framework for visual saliency estimation. We have developed a new spatio-temporal visual frontend based on biologically inspired 3D Gabor filters, which is applied on both the luminance and the color streams and produces spatio-temporal energy maps. These volumes are fused for computing a single saliency map and can detect spatio-temporal phenomena that static saliency models cannot find. We also provide a new movie database with eye-tracking annotation. We have evaluated our spatio-temporal saliency model on the widely used CRCNS-ORIG database as well as our new database using different fusion schemes and feature sets. The proposed spatio-temporal computational framework incorporates many ideas based on psychological evidences and yields significant improvements on spatio-temporal saliency estimation.