Deep belief echo-state network and its application to time series prediction

Deep belief network (DBN) has attracted many attentions in time series prediction. However, the DBN-based methods fail to provide favorable prediction results due to the congenital defects of the back-propagation method, such as slow convergence and local optimum. To address the problems, we propose a deep belief echo-state network (DBEN) for time series prediction. In the new architecture, DBN is employed for feature learning in an unsupervised fashion, which can effectively extract hierarchical data features. An innovative regression layer, embedding an echo-state learning mechanism instead of the traditional back-propagation method, is built on top of DBN for supervised prediction. To our best knowledge, this is the first paper that applies the echo state network methodology to deep learning. The resulted model, combining the merits of DBN and ESN, provides a more robust alternative to conventional deep neural networks for the superior prediction capacity. Extensive experimental results show that our DBEN can achieve a significant enhancement in the prediction performance, learning speed, and short-term memory capacity.