Attention pooling-based convolutional neural network for sentence modelling

Convolutional neural network has been proven to be a powerful semantic composition model for modelling sentences. A standard convolutional neural network usually consists of several convolutional and pooling layers at the bottom of a linear or non-linear classifier. In this paper, a new pooling scheme termed Attention Pooling is proposed to retain the most significant information at the pooling stage. An intermediate sentence representation generated by the bidirectional long short-term memory is used as a reference for local representations produced by the convolutional layer to obtain attention weights. The sentence representation is formed by combining local representations using obtained attention weights. The intermediate sentence representation is used as an input to the top classifier as well in the testing phase. The salient features of the proposed attention pooling-based convolutional neural network are: (1) The model can be trained end-to-end with limited hyper-parameters; (2) Comprehensive information is extracted by the new pooling scheme and the combination of the convolutional layer and the bidirectional long-short term memory; (3) The model can implicitly separate the sentences from different classes. Experimental results demonstrate that the new model outperforms the state-of-the-art approaches on seven benchmark datasets for text classification. The learning capability of the proposed method is greatly improved and the classification accuracy is even enhanced significantly by over 2% on some datasets. The robustness of the proposed model is evidenced by some statistical tests.