An Information Theoretic Approach to Constructing Machine Learning Criteria

Selecting a learning criterion is a constituent part of a machine learning problem statement requiring both accounting its adequacy to the data available and practical suitability of implementation. The paper presents an approach to the machine learning in accordance to information-theoretic criteria that are derived basing on the Rényi entropy of an arbitrary order. Meanwhile, a parameterized description of the machine learning is utilized combined with a corresponding technique of estimation of mutual information constructed basing on the Rényi entropies. This leads, finally, to a problem of the finite dimensional optimization to be solved by a suitable technique. The consideration proposed is preceded by a thorough review of existing information theoretic and entropy based approaches to the machine learning. The paper has been supported by a grant of the Russian Foundation for Basic Researches (RFBR): project 12-08-01205-a.