Transfer metric learning for action similarity using high-level semantics

• We analyze the benefits of using high-level semantics for transfer metric learning.
• We present a hierarchical model based on the embedded structure in attribute space.
• We propose a novel decorrelated normalized space (DNS) for transfer metric learning.
• We test using a challenging scheme where the target is more diverse than the source.
• Representation provides superior performance and DNS improves transfer effectiveness.

The goal of transfer learning is to exploit previous experiences and knowledge in order to improve learning in a novel domain. This is especially beneficial for the challenging task of learning classifiers that generalize well when only few training examples are available. In such a case, knowledge transfer methods can help to compensate for the lack of data. The performance and robustness against negative transfer of these approaches is influenced by the interdependence between knowledge representation and transfer type. However, this important point is usually neglected in the literature; instead the focus lies on either of the two aspects. In contrast, we study in this work the effect of various high-level semantic knowledge representations on different transfer types in a novel generic transfer metric learning framework. Furthermore, we introduce a hierarchical knowledge representation model based on the embedded structure in the semantic attribute space. The evaluation of the framework on challenging transfer settings in the context of action similarity demonstrates the effectiveness of our approach compared to state-of-the-art.