The Uncertainty Area Metric: a Method for Comparing Learning Machines on What They Don't Know

Schaffer and Land14 described a method whereby a machine intelligence (MI) process can “know what it doesn't know.” In this paper, the concept is illustrated by three examples: the GRNN oracle ensemble method that combines multiple SVM classifiers for detecting Alzheimer's type dementia using features automatically extracted from a speech sample, an Evolutionary Programming and Adaptive Boosting hybrid and a Generalized Regression Neural Network hybrid for classifying breast cancer. The authors assert it is (1) applicable quite directly to a great many other learning classifier systems, and (2) provides an intuitive approach to comparing the performance of different classifiers on a given task using the size of the “area of uncertainty” as a measure of performance metric. This paper provides support for these assertions by describing the steps needed to apply it to a previously published study of breast cancer benign / malignancy prediction, and then illustrates how this “area of uncertainty” may be computed, which is a work in progress, using the GRNN oracle results and a resultant Bayesian network from the Alzheimer's speech research study.