Evaluation and amelioration of computer-aided diagnosis with artificial neural networks utilizing small-sized sample sets

Concerns about the specificity and reliability of artificial neural networks (ANNs) impede further application of ANNs in medicine. This is particularly true when developing computer-aided diagnosis (CAD) tools using ANNs for orphan diseases and emerging research areas where only a small-sized sample set is available. It is unreasonable to claim one ANN's performance as better than another simply on the basis of a single output without considering possible output variability due to factors including data noise and ANN training protocols. In this paper, a bootstrap resampling method is proposed to quantitatively analyze ANN output reliability and changing performance as the sample data and training protocols are varied. The method is tested in the area of feature classification for analysis of masses detected on mammograms. Our experiments show that ANNs performance, measured in terms of the area under the receiver operating characteristic (ROC) curve, is not a fixed value, but follows a distribution function sensitive to many factors. We demonstrate that our approach to determining the bootstrap estimates of confidence intervals (CIs) and prediction intervals (PIs) can be used to assure optimal performance in terms of ANN model configuration. We also show that the unintentional inclusion of data noise, which biases ANN results in small task-specific databases, can be accurately detected via the bootstrap estimates.

► Output of ANN built with small-sized sample sets can be subject to large variance. ► Bootstrap approach provides quantitative measurements without cost of large datasets. ► Bootstrap estimates of CIs and PIs can be used to assure optimal ANN performance. ► Bootstrap estimates can accurately detect inconsistent sample data. ► We apply bootstrap to ANNs in a clinical CAD application on small medical datasets.