کد مقاله کد نشریه سال انتشار مقاله انگلیسی نسخه تمام متن
515358 866998 2015 18 صفحه PDF دانلود رایگان
عنوان انگلیسی مقاله ISI
Statistical comparisons of non-deterministic IR systems using two dimensional variance
موضوعات مرتبط
مهندسی و علوم پایه مهندسی کامپیوتر نرم افزارهای علوم کامپیوتر
پیش نمایش صفحه اول مقاله
Statistical comparisons of non-deterministic IR systems using two dimensional variance
چکیده انگلیسی


• We propose methods to compare non-deterministic IR systems.
• We show pitfalls in using standard significance tests to compare such systems.
• We verify the applicability of proposed methods using simulations and a case study.
• We show how to compare a non-deterministic IR system for equivalent effectiveness.

Retrieval systems with non-deterministic output are widely used in information retrieval. Common examples include sampling, approximation algorithms, or interactive user input. The effectiveness of such systems differs not just for different topics, but also for different instances of the system. The inherent variance presents a dilemma – What is the best way to measure the effectiveness of a non-deterministic IR system? Existing approaches to IR evaluation do not consider this problem, or the potential impact on statistical significance. In this paper, we explore how such variance can affect system comparisons, and propose an evaluation framework and methodologies capable of doing this comparison.Using the context of distributed information retrieval as a case study for our investigation, we show that the approaches provide a consistent and reliable methodology to compare the effectiveness of a non-deterministic system with a deterministic or another non-deterministic system. In addition, we present a statistical best-practice that can be used to safely show how a non-deterministic IR system has equivalent effectiveness to another IR system, and how to avoid the common pitfall of misusing a lack of significance as a proof that two systems have equivalent effectiveness.

ناشر
Database: Elsevier - ScienceDirect (ساینس دایرکت)
Journal: Information Processing & Management - Volume 51, Issue 5, September 2015, Pages 677–694
نویسندگان
, , , , ,