Improving mixing rate with tempered transition for learning restricted Boltzmann machines

• A new training method for Restricted Boltzmann Machines (RBMs) is proposed.
• The new method is based on Tempered Transition which is used to sample from RBMs.
• Tempered Transition has strong ability for handling multimodal distributions.
• Tempered Transition based training method can achieve better mixing rate than the existing methods.
• Tempered Transition based training method can help approximate the likelihood gradient.

Recently, as the building block of deep generative models such as Deep Belief Networks (DBNs), Restricted Boltzmann Machines (RBMs) have attracted much attention. RBM is a Markov Random Field (MRF) associated with a bipartite undirected graph which is famous for powerful expression and tractable inference. While training an RBM, we need to sample from the model. The larger the mixing rate is, the smaller the bias of the samples is. However, neither Gibbs sampling based training methods such as Contrastive Divergence (CD) nor Parallel Tempering based training methods can achieve satisfying mixing rate, which causes poor rendering of the diversity of the modes captured by these trained models. This property may hinder the existing methods to approximate the likelihood gradient. In order to alleviate this problem, we attempt to introduce Tempered Transition, an advanced tempered Markov Chain Monte Carlo method, into training RBMs to replace Gibbs sampling or Parallel Tempering for sampling from RBMs. Experimental results show that our proposed method outperforms the existing methods to achieve better mixing rate and to help approximate the likelihood gradient.