Effect of classifier selection, reference sample size, reference class distribution and scene heterogeneity in per-pixel classification accuracy using 26 Landsat sites

A major issue in land cover mapping is classifier selection. Here we investigated classifier performance under different sample sizes, reference class distribution, and scene complexities. Twenty six 10 km × 10 km blocks with complete reference information across the continental US are used. Per-pixel classification took place using six spectral bands from Landsat imagery. The tested classifiers included Naïve Bayes (NB), Support Vector Machine (SVM), K-Nearest Neighbor (KNN), Bootstrap-aggregation ensemble of decision trees (BagTE), artificial neural network up to 2 hidden layers, and deep neural network (DNN) up to 3 hidden layers. For the entire block, our accuracy assessment indicated that all classifiers, with the exception of NB (a Maximum Likelihood variant), performed similarly. However, when we concentrated on edge pixels (pixels at the border of adjacent land cover classes), it was clear that the SVM and KNN offer considerable accuracy advantages, especially for larger reference datasets. Because of their relatively low execution times SVM and KNN would be recommended for classifications using Landsat's spectral inputs and Anderson's 11-level classification scheme. However, both SVM and KNN demonstrated substantial accuracy degradation during the parameter grid search. For this reason, an exhaustive parameter optimization process is suggested. While the ANN and DNN neural network variants did not perform as well, their performance may have been restricted by the lack of rich contextual information in our simple six band per-pixel input space. The effect of class distribution in the training dataset was also evident on the calculated accuracy metric. Gradual accuracy degradation as edge pixel presence increased was also observed. Future work could focus on data-rich classification problems such as change detection using Landsat stacks or expand in high spectral or spatial resolution sensors.