A comparative study of data fusion for RGB-D based visual recognition

• This study investigates key aspects for RGB-D based visual recognition, including the data fusion schemes and the classifiers.
• This study conducts a comparative study for evaluating how the recognition performance is affected.
• This study is the first work to explicitly address the fusion issue for RGB-D data with deep learning.
• This study can serve as useful guidance for developing visual recognition systems in the related application fields.

Data fusion from different modalities has been extensively studied for a better understanding of multimedia contents. On one hand, the emergence of new devices and decreasing storage costs cause growing amounts of data being collected. Though bigger data makes it easier to mine information, methods for big data analytics are not well investigated. On the other hand, new machine learning techniques, such as deep learning, have been shown to be one of the key elements in achieving state-of-the-art inference performances in a variety of applications. Therefore, some of the old questions in data fusion are in need to be addressed again for these new changes. These questions are: What is the most effective way to combine data for various modalities? Does the fusion method affect the performance with different classifiers? To answer these questions, in this paper, we present a comparative study for evaluating early and late fusion schemes with several types of SVM and deep learning classifiers on two challenging RGB-D based visual recognition tasks: hand gesture recognition and generic object recognition. The findings from this study provide useful policy and practical guidance for the development of visual recognition systems.