Hierarchical spatiotemporal feature extraction using recurrent online clustering

• Deep learning architecture for capturing spatiotemporal dependencies in observations.
• Novel incremental recurrent clustering algorithm that identifies temporal saliencies.
• A revised DeSTIN architecture that can be efficiently implemented on GPU platforms.

Deep machine learning offers a comprehensive framework for extracting meaningful features from complex observations in an unsupervised manner. The majority of deep learning architectures described in the literature primarily focus on extracting spatial features. However, in real-world settings, capturing temporal dependencies in observations is critical for accurate inference. This paper introduces an enhancement to DeSTIN – a compositional deep learning architecture in which each layer consists of multiple instantiations of a common node – that learns to represent spatiotemporal patterns in data based on a novel recurrent clustering algorithm. Contrary to mainstream deep architectures, such as deep belief networks where layer-by-layer training is assumed, each of the nodes in the proposed architecture is trained independently and in parallel. Moreover, top-down and bottom-up information flows facilitate rich feature formation. A semi-supervised setting is demonstrated achieving state-of-the-art results on the MNIST classification benchmarks. A GPU implementation is discussed further accentuating the scalability properties of the proposed framework.