Alignment-consistent recursive neural networks for bilingual phrase embeddings

Learning semantic representations of bilingual phrases is very important for statistical machine translation to overcome data sparsity and exploit semantic information. In this paper, we consider word alignments as a semantic bridge between the source and target phrases, and propose two neural networks based on the conventional recursive autocoder, which exploit word alignments to generate alignment-consistent bilingual phrase structures: One is Alignment Enhanced Recursive Autoencoder that incorporates a word-alignment-related error into the final objective function; The other is Alignment Guided Recursive Neural Network which treats word alignments as direct signals to guide phrase structure constructions. Then, we further establish the semantic correspondences between the source and target nodes of the generated bilingual phrase structures via word alignments. By jointly minimizing recursive autoencoder reconstruction errors, structural alignment consistency errors and cross-lingual reconstruction errors, our model not only generates alignment-consistent phrase structures, but also captures different levels of semantic correspondences within bilingual phrases. Experiments on the NIST Chinese-English translation task show that our model achieves significant improvements over the baseline.