کد مقاله | کد نشریه | سال انتشار | مقاله انگلیسی | نسخه تمام متن |
---|---|---|---|---|
13435965 | 1842988 | 2020 | 20 صفحه PDF | دانلود رایگان |
عنوان انگلیسی مقاله ISI
Effectiveness evaluation without human relevance judgments: A systematic analysis of existing methods and of their combinations
دانلود مقاله + سفارش ترجمه
دانلود مقاله ISI انگلیسی
رایگان برای ایرانیان
کلمات کلیدی
موضوعات مرتبط
مهندسی و علوم پایه
مهندسی کامپیوتر
نرم افزارهای علوم کامپیوتر
پیش نمایش صفحه اول مقاله
![عکس صفحه اول مقاله: Effectiveness evaluation without human relevance judgments: A systematic analysis of existing methods and of their combinations Effectiveness evaluation without human relevance judgments: A systematic analysis of existing methods and of their combinations](/preview/png/13435965.png)
چکیده انگلیسی
In test collection based evaluation of retrieval effectiveness, it has been suggested to completely avoid using human relevance judgments. Although several methods have been proposed, their accuracy is still limited. In this paper we present two overall contributions. First, we provide a systematic comparison of all the most widely adopted previous approaches on a large set of 14 TREC collections. We aim at analyzing the methods in a homogeneous and complete way, in terms of the accuracy measures used as well as in terms of the datasets selected, showing that considerably different results may be achieved considering different methods, datasets, and measures. Second, we study the combination of such methods, which, to the best of our knowledge, has not been investigated so far. Our experimental results show that simple combination strategies based on data fusion techniques are usually not effective and even harmful. However, some more sophisticated solutions, based on machine learning, are indeed effective and often outperform all individual methods. Moreover, they are more stable, as they show a smaller variation across datasets. Our results have the practical implication that, when trying to automatically evaluate retrieval effectiveness, researchers should not use a single method, but a (machine-learning based) combination of them.
ناشر
Database: Elsevier - ScienceDirect (ساینس دایرکت)
Journal: Information Processing & Management - Volume 57, Issue 2, March 2020, 102149
Journal: Information Processing & Management - Volume 57, Issue 2, March 2020, 102149
نویسندگان
Kevin Roitero, Andrea Brunello, Giuseppe Serra, Stefano Mizzaro,