Maxout neurons for deep convolutional and LSTM neural networks in speech recognition

Deep neural networks (DNNs) have achieved great success in acoustic modeling for speech recognition. However, DNNs with sigmoid neurons may suffer from the vanishing gradient problem during training. Maxout neurons are promising alternatives to sigmoid neurons. The activation of a maxout neuron is obtained by selecting the maximum value within a local region, which results in constant gradients during the training process. In this paper, we combine the maxout neurons with two popular DNN structures for acoustic modeling, namely the convolutional neural network (CNN) and the long short-term memory (LSTM) recurrent neural network (RNN). The optimal network structures and training strategies for the models are explored. Experiments are conducted on the benchmark data sets released under the IARPA Babel Program. The proposed models achieve 2.5-6.0% relative improvements over their corresponding CNN or LSTM RNN baselines across six language collections. The state-of-the-art results on these data sets are achieved after system combination.