Information Theoretic Interpretation of Error Criteria

Error criteria (or error cost functions) play significant roles in statistical estimation problems. In this paper, we study error criteria from the viewpoint of information theory. The relationships between error criteria and error's entropy criterion are investigated. It is shown that an error criterion is equivalent to the error's entropy criterion plus a Kullback-Leibler information divergence (KL-divergence). Based on this result, two important properties of the error criteria are proved. Particularly, the optimum error criterion can be interpreted via the meanings of entropy and KL-divergence. Furthermore, a novel approach is proposed for the choice of p-power error criteria, in which a KL-divergence based cost is minimized. The proposed method is verified by Monte Carlo simulation experiments.