A methodology for comparing classification methods through the assessment of model stability and validity in variable selection

Classification analysis utilizes features for separating observations into distinct groups for decision-making purposes. This study provides a systematic design for comparing the performance of six classification methods using Monte Carlo simulations and illustrates that the variable selection process is integral in comparing methodologies to ensure minimal bias, enhanced stability, and optimize performance. We quantify the variable selection bias and show that, for sufficiently large samples, this bias is minimized so that methods can be compared. We address topics relevant to model building and provide prescriptions for future comparisons so as to build a body of evidence for recommending their use.

► We describe a systematic design for comparing the performance of six classification methods. ► We show that the variable selection process is integral in comparing methods to ensure minimal bias and optimal performance. We show that, for sufficiently large samples, this bias is minimized so that methods can be compared. ► Previous sample size recommendations are insufficient. ► We address topics relevant to model building and provide prescriptions for future comparisons so as to build a body of evidence for recommending their use.