Statistical comparison of image fusion algorithms: Recommendations

Pixel-level image fusion has been applied in a variety of applications, including multi-modal medical imaging, remote sensing, industrial inspection, video surveillance, and night vision etc. Various algorithms are being proposed for numerous applications which requires a comprehensive method of assessment to discern which methods provide decision support. Currently, the validation or assessment of newly proposed algorithms is done either subjectively or objectively. A subjective assessment is costly and affected by a number of factors that are difficult to control. On the other hand, an objective assessment is carried out with a fusion performance metric which is defined to evaluate the effectiveness and/or efficiency of the fusion operation. There are a number of fusion metrics proposed for fusion processes taking different perspectives. Most image fusion research presents a comparison of the proposed and existing fusion algorithms with selected fusion metric(s) over multiple image data sets. The proposed algorithm advantage is justified by the relative difference with the best or better metric values. However, the statistical significance of such difference is unknown leading to a misperception of the quantitative differences between methods. This paper proposes the use of non-parametric statistical analysis for comparisons of fusion algorithms along with the Image fusion Toolbox Employing Significance Testing (ImTEST). Strategies to use different tests in varied scenarios are presented and recommended. Experiments with recently published algorithms demonstrate the necessity to adopt the statistical comparison to establish a baseline for image fusion research.