Benchmarking reputation systems: A quantitative verification approach

•Proposing a novel and formal robustness measure for reputation systems (RSs).•Benchmarking a number of well-known RSs against basic and the worst case attacks.•Exploring the strength and weakness of the RSs in different environment.•Ranking and classification of the RSs.•Introducing RepSyFire as an open source tool for verification of RSs.

Trust and reputation systems are classes of decision support tools which help detecting malicious behavior based on collecting ratings and opinions. Despite their advantages, these systems are vulnerable to some kinds of attacks in which the attacker can deceive the system using sequences of misleading behaviors. Robustness of reputation systems against these attacks are frequently investigated in the literature. However the existing works usually evaluate the robustness using a qualitative simulation method. Lack of a formal evaluation method and a quantitative measure of robustness make it hard to extend the results and to compare the systems precisely. This paper proposes a quantitative robustness measure for reputation systems based on a formal verification approach. Using the robustness measure and the verification method, a comprehensive benchmarking of a number of well-known reputation systems is presented which includes evaluation of the systems against basic and the worst case attacks. The results are used for ranking and classifying the systems. The studies show that robustness is not an absolute feature of a reputation model, but it also depends on the properties of the environment. The benchmarking results have been also used to indicate the proper environment for each class of systems/attacks.