VIRTUE: A visual tool for information retrieval performance evaluation and failure analysis

• Visual tool for conducting performance and failure analysis.
• Integration between Information Retrieval Evaluation and Visual Analytics methodologies.
• Easing the interaction and interpretation of evaluation metric curves.
• Highlighting critical areas of rankings to quickly inspect and detect causes of failure.
• Exploration of the IR experimental data at both topic and experiment levels.

Objective: Information Retrieval (IR) is strongly rooted in experimentation where new and better ways to measure and interpret the behavior of a system are key to scientific advancement. This paper presents an innovative visualization environment: Visual Information Retrieval Tool for Upfront Evaluation (VIRTUE), which eases and makes more effective the experimental evaluation process.Methods: VIRTUE supports and improves performance analysis and failure analysis.Performance analysis: VIRTUE offers interactive visualizations based on well-known IR metrics allowing us to explore system performances and to easily grasp the main problems of the system.Failure analysis: VIRTUE develops visual features and interaction, allowing researchers and developers to easily spot critical regions of a ranking and grasp possible causes of a failure.Results: VIRTUE was validated through a user study involving IR experts. The study reports on (a) the scientific relevance and innovation and (b) the comprehensibility and efficacy of the visualizations.Conclusion: VIRTUE eases the interaction with experimental results, supports users in the evaluation process and reduces the user effort.Practice: VIRTUE will be used by IR analysts to analyze and understand experimental results.Implications: VIRTUE improves the state-of-the-art in the evaluation practice and integrates visualization and IR research fields in an innovative way.