Tunable discounting and visual exploration for language models

A language model is fundamental to many applications in natural language processing. Most language models are trained on a large amount of dataset and difficult to be adapted to other domains which may have only a small dataset available. Tuning discounting parameters for smoothing is one way to adapt language models for a new domain. In this work, we present novel language models based on tunable discounting mechanisms. The language models are trained on a large dataset, but their discounting parameters can be tuned to a target dataset afterwards. We explore tunable discounting and polynomial discounting functions based on the modified Kneser-Ney (mKN) models. Specifically, we propose the tunable mKN (TmKN) model, polymomial discounting mKN (PmKN) model, and tunable and polynomial discounting mKN (TPmKN) model. We test our proposed models and compared with the mKN model, improved KN model, and the tunable mKN with the interpolation model (mKN + interp). With the implementation, our language models achieve perplexity improvements in both in-domain and out-of-domain evaluation. Experimental results indicate that our new models significantly outperform the baseline model and our models are especially suitable for adapting to new domains. In addition, we use the visualization technique to depict the relationship between parameter settings and the language model performances for guiding our parameter optimization process. The exploratory visual analysis is then used to examine the performance of the proposed language models which will reveal the strength and characteristic of the models.