The AmericaView classification methods accuracy comparison project: A rigorous approach for model selection

•We compared overall accuracy of six classification methods using 30 datasets.•Accuracies were also compared for spectral band data only and for rare classes.•Random forest had higher mean overall accuracies than other methods tested.•The highest accuracies were dataset dependent; no method was best for all purposes.

Evaluation of classification methods, whether in connection with the development of new methods or in an application setting, has been hampered by the lack of availability of adequate data and an approach for comparisons. We collected 30 mostly moderate-resolution, multispectral datasets to enable statistically rigorous comparisons of methods and have made those datasets available for other researchers. We developed a methodological approach to comparing classification methods and demonstrated the approach using six methods, C5.0, classification tree analysis, logistic model trees, multivariate adaptive regression splines, random forest, and support vector machines. We also demonstrated how these data and this approach can be used to address specific questions in addition to overall accuracy performance, including the relative effects of using derived components and ancillary data and the relative success in classifying rare classes. Most methods performed best by at least one metric with at least one dataset. Therefore, although random forest on average performed statistically significantly better than the other methods tested, we do not recommend this method as the sole option currently in remote sensing. Rather, our results suggest that remote sensing analysts should evaluate multiple methods with respect to any classification project, which can be accomplished through statistical software packages.