Multispectral pedestrian detection based on deep convolutional neural networks

Vision-based pedestrian detection that can last all day is crucial in advanced driver-assistance systems (ADAS), autonomous vehicles and video surveillance. Based on the fact that the wavelength of human body radiation falls around 9.3 μm, thermal images have distinctive advantages for pedestrian detection in the nighttime. With the recent success of convolutional neural networks (CNNs) in the vision community, how to properly explore information in color and thermal images using CNNs-based methods attracts the attention of researchers. Previous CNN-based multispectral pedestrian detectors focus on the design of network architectures. The main contributions of this paper are as follows. First, pixel-level image fusion are also extensively evaluated based on a state-of-the-art CNNs-based framework in the daytime and nighttime for pedestrian detection. Second, the effective strategies to combine pixel-level fusion methods and CNN-fusion architectures are studied based on extensive experimental results. Two combination strategies are designed and an exhaustive experimental analysis is performed to evaluate different combinations for all-day pedestrian detection. Extensive results based on a multispectral pedestrian benchmark show that some pixel-level image fusion methods can achieve similar or even better performance than CNN-fusion architectures, which emphasizes the importance of pixel-level fusion in CNN-based pedestrian detectors. The combination of both can usually more properly exploit multispectral information and further boost detection performance.