Assessing reference dataset representativeness through confidence metrics based on information density

Land cover maps obtained from classification of remotely sensed imagery provide valuable information in numerous environmental monitoring and modeling tasks. However, many uncertainties and errors can directly or indirectly affect the quality of derived maps. This work focuses on one key aspect of the supervised classification process of remotely sensed imagery: the quality of the reference dataset used to develop a classifier. More specifically, the representative power of the reference dataset is assessed by contrasting it with the full dataset (e.g. entire image) needing classification. Our method is applicable in several ways: training or testing datasets (extracted from the reference dataset) can be compared with the full dataset. The proposed method moves beyond spatial sampling schemes (e.g. grid, cluster) and operates in the multidimensional feature space (e.g. spectral bands) and uses spatial statistics to compare information density of data to be classified with data used in the reference process. The working hypothesis is that higher information density, not in general but with respect to the entire classified image, expresses higher confidence in obtained results. Presented experiments establish a close link between confidence metrics and classification accuracy for a variety of image classifiers namely maximum likelihood, decision tree, Backpropagation Neural Network and Support Vector Machine. A sensitivity analysis demonstrates that spatially-continuous reference datasets (e.g. a square window) have the potential to provide similar classification confidence as typically-used spatially-random datasets. This is an important finding considering the higher acquisition costs for randomly distributed datasets. Furthermore, the method produces confidence maps that allow spatially-explicit comparison of confidence metrics within a given image for identification of over- and under-represented image portions. The current method is presented for individual image classification but, with sufficient evaluation from the remote sensing community it has the potential to become a standard for reference dataset reporting and thus allowing users to assess representativeness of reference datasets in a consistent manner across different classification tasks.