Visible-infrared fusion schemes for road obstacle classification

• We aimed VIS-IR probabilistic fusion schemes at features, kernels and matching scores.
• We use an adapted static or dynamic weighting of VIS and IR modalities.
• The problem of a multi-class road obstacle classification was approached using SVMs.
• Different families of features were combined.
• Different features selection algorithms were tested.

In this article we propose different fusion schemes using information provided by VISible (VIS) and InfraRed (IR) images for road obstacle SVM (Support Vector Machine)-based classification. Three probabilistic approaches for the fusion of VIS and IR images are presented. The early fusion at the feature level yields a bimodal feature vector integrating both VIS and IR data, used to feed an SVM-based classifier. An intermediate fusion at the kernel level combines two different monomodal kernels in order to obtain a particularly flexible Bimodal Kernel (BK), we believe more appropriate for heterogeneous VIS and IR data classification with SVM. The late fusion combines matching scores provided by VIS and IR obstacle recognition modules in order to improve the system performance. An important advantage of these fusion schemes is their capability to adapt to the environmental illumination changes and specific weather conditions due to a modality weighting parameter which allows to adjust the decision of the system according to the relative importance of the VIS and IR modalities. Experiments performed on the TetraVision image database showed that all our fusion-based obstacle classifiers outperform both monomodal VIS and IR obstacle recognizers. The matching score fusion with a dynamic weighting scheme provides the best results compared with both early and intermediate fusion schemes using static modality weights. The BK scheme we propose for VIS–IR fusion would need a greater and better balanced database for learning improvement, since the BK has much more hyper-parameters to be simultaneously optimized than the matching-score fusion.