Decision making matters: A better way to evaluate trust models

Trust models are mechanisms that predict behavior of potential interaction partners. They have been proposed in several domains and many advances in trust formation have been made recently. The question of comparing trust models, however, is still without a clear answer. Traditionally, authors set up ad hoc experiments and present evaluation results that are difficult to compare – sometimes even interpret – in the context of other trust models. As a solution, the community came up with common evaluation platforms, called trust testbeds. In this paper we expose shortcomings of evaluation models that existing testbeds use; they evaluate trust models by combining them with some ad hoc decision making mechanism and then evaluate the quality of trust-based decisions. They assume that if all trust models use the same decision making mechanism, the mechanism itself becomes irrelevant for the evaluation. We hypothesized that the choice of decision making mechanism is in fact relevant. To test our claim we built a testbed, called Alpha testbed, that can evaluate trust models either with or without decision making mechanism. With it we evaluated five well-known trust models using two different decision making mechanisms. The results confirm our hypothesis; the choice of decision making mechanisms influences the performance of trust models. Based on our findings, we recommend to evaluate trust models independently of the decision making mechanism – and we also provide a method (and a tool) to do so.