A modified version of Helmholtz machine by using a Restricted Boltzmann Machine to model the generative probability of the top layer

The Helmholtz machine is an unsupervised deep neural network with different bottom-up recognition weights and top-down generative weights, which attempts to build probability density models of sensory inputs. The recognition weights are used to determine the recognition probability of each unit from bottom layer to top layer and the generative weights are used to determine the generative probability of each unit from top layer to bottom layer. The model parameters can be gained by minimizing the sum of the Kullback-Leibler divergence between generative and recognition distributions of all units. In this paper, we proposed a modified Helmholtz machine by adding an additional hidden layer on the top layer of the Helmholtz machine, which is used to model the generative probability of the top layer. The additional added hidden layer provides 'complementary prior' to the original top layer and can eliminate the 'explaining away effects' to make the Helmholtz machine fitting sensory inputs much better. The experimental results of new algorithm on various data sets show that the modified Helmholtz machine learns better generative models.