Evaluating evaluation metrics based on the bootstrap

No abstract is available for this paper.

• Publication year

  2006

• Venue

  Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

• Publication date

  2006-08-06

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/1148170.1148261
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• On the reliability of information retrieval metrics based on graded relevance

  2007influential reference
• An Introduction to the Bootstrap

  2007cited by this paper
• Toshiba BRIDJE at NTCIR-5 CLIR: Evaluation using Geometric Means

  2005cited by this paper
• Revisiting the effect of topic set size on retrieval error

  2005cited by this paper
• The Effect of Topic Sampling on Sensitivity Comparisons of Information Retrieval Metrics

  2005cited by this paper
• Binary and graded relevance in IR evaluations--Comparison of the effects on ranking of IR systems

  2005influential reference
• Information retrieval system evaluation: effort, sensitivity, and reliability

  2005influential reference
• On Effectiveness Measures and Relevance Functions in Ranking INEX Systems

  2005influential reference
• Binary and graded relevance in IR evaluations-Comparison of the effects on ranking of IR systems

  2005influential reference
• Overview of the TREC 2004 Robust Retrieval Track

  2004influential reference
• The effect of topic set size on retrieval experiment error

  2002influential reference
• Cumulated gain-based evaluation of IR techniques

  2002influential reference
• The Insignificance of Statistical Significance Testing

  1999cited by this paper
• Statistical inference in retrieval effectiveness evaluation

  1997influential reference
• Using statistical testing in the evaluation of retrieval experiments

  1993cited by this paper
• Statistical Inference

  1990cited by this paper

• Open-Source LLM-based Relevance Assessment vs. Highly Reliable Manual Relevance Assessment: A Case Study

  2025cites this paper
• OrdRankBen: A Novel Ranking Benchmark for Ordinal Relevance in NLP

  2025cites this paper
• Performance Consistency of Learning Methods for Information Retrieval Tasks

  2025cites this paper
• FiGVCL: Fine-Grained Benchmark and Method for Video Copy Localization

  2025cites this paper
• Measuring Group Fairness in Web Search: AWRF or GFR?

  2025cites this paper
• Semi-Supervised Learned Autoencoder for Classification of Events in Distributed Fibre Acoustic Sensors

  2025cites this paper
• Sim4IA-Bench: A User Simulation Benchmark Suite for Next Query and Utterance Prediction

  2025cites this paper
• Handling new users and items: a comparative study of inductive recommenders

  2025cites this paper
• Team OpenWebSearch at CLEF 2024: QuantumCLEF

  2024cites this paper
• UFO: a Unified and Flexible Framework for Evaluating Factuality of Large Language Models

  2024influential citation
• Decoy Effect in Search Interaction: Understanding User Behavior and Measuring System Vulnerability

  2024cites this paper
• Are LLMs Effective Negotiators? Systematic Evaluation of the Multifaceted Capabilities of LLMs in Negotiation Dialogues

  2024cites this paper
• Bootstrapped nDCG Estimation in the Presence of Unjudged Documents

  2023influential citation
• How Well do Offline Metrics Predict Online Performance of Product Ranking Models?

  2023cites this paper
• A Reference-Dependent Model for Web Search Evaluation: Understanding and Measuring the Experience of Boundedly Rational Users

  2023cites this paper
• The Impact of Judgment Variability on the Consistency of Offline Effectiveness Measures

  2023cites this paper
• Joint upper & expected value normalization for evaluation of retrieval systems: A case study with Learning-to-Rank methods

  2023cites this paper
• Perspectives on Large Language Models for Relevance Judgment

  2023cites this paper
• The BETTER Cross-Language Datasets

  2023cites this paper
• IR Systems Evaluation via Generalized Linear Models

  2023cites this paper
• A Meta-Evaluation of C/W/L/A Metrics: System Ranking Similarity, System Ranking Consistency and Discriminative Power

  2023cites this paper
• A Versatile Framework for Evaluating Ranked Lists in Terms of Group Fairness and Relevance

  2022cites this paper
• sMARE: a new paradigm to evaluate and understand query performance prediction methods

  2022cites this paper
• Re-thinking Knowledge Graph Completion Evaluation from an Information Retrieval Perspective

  2022influential citation
• Evaluating Top-N Recommendations Using Ranked Error Approach: An Empirical Analysis

  2022cites this paper
• Detecting Significant Differences Between Information Retrieval Systems via Generalized Linear Models

  2022cites this paper
• IR Evaluation and Learning in the Presence of Forbidden Documents

  2022influential citation
• Joint Upper & Lower Bound Normalization for IR Evaluation

  2022influential citation
• Towards Unified Metrics for Accuracy and Diversity for Recommender Systems

  2021influential citation
• A Fast and Exact Randomisation Test for Comparing Two Systems with Paired Data

  2021cites this paper
• Meta-evaluation of Conversational Search Evaluation Metrics

  2021cites this paper
• On the Two-Sample Randomisation Test for IR Evaluation

  2021cites this paper
• A Closer Look at Evaluation Measures for Ordinal Quantification

  2021cites this paper
• POSSCORE: A Simple Yet Effective Evaluation of Conversational Search with Part of Speech Labelling

  2021cites this paper
• Covid-19 detection using chest X-rays: is lung segmentation important for generalization?

  2021cites this paper
• Incorporating Query Reformulating Behavior into Web Search Evaluation

  2021cites this paper
• Evaluating Relevance Judgments with Pairwise Discriminative Power

  2021influential citation
• Evaluating Evaluation Measures for Ordinal Classification and Ordinal Quantification

  2021cites this paper
• A user-centred approach to information retrieval

  2021cites this paper
• On the Quality of the TREC-COVID IR Test Collections

  2021cites this paper
• Assessing ranking metrics in top-N recommendation

  2020influential citation
• Assessing Top- Preferences

  2020cites this paper
• Diversity and novelty in biomedical information retrieval

  2020cites this paper
• Models Versus Satisfaction: Towards a Better Understanding of Evaluation Metrics

  2020cites this paper
• The Impact of Negative Relevance Judgments on NDCG

  2020cites this paper
• Corpus Bootstrapping for Assessment of the Properties of Effectiveness Measures

  2020cites this paper
• User behavior modeling for Web search evaluation

  2020cites this paper
• On Target Item Sampling in Offline Recommender System Evaluation

  2020cites this paper
• Exploring Topic Difficulty in Information Retrieval Systems Evaluation

  2019cites this paper
• NII Testbeds and Community for Information Access Research: 14th International Conference, NTCIR 2019, Tokyo, Japan, June 10–13, 2019, Revised Selected Papers

  2019influential citation
• Studying Online and Offline Evaluation Measures: A Case Study Based on the NTCIR-14 OpenLiveQ-2 Task

  2019cites this paper
• From Multilingual to Multimodal: The Evolution of CLEF over Two Decades

  2019cites this paper
• How do interval scales help us with better understanding IR evaluation measures?

  2019cites this paper
• On the discriminative power of hyper-parameters in cross-validation and how to choose them

  2019cites this paper
• Improving Web Image Search with Contextual Information

  2019cites this paper
• The Evolution of Cranfield

  2019cites this paper
• Information Retrieval Evaluation in a Changing World: Lessons Learned from 20 Years of CLEF

  2019cites this paper
• DCU at the NTCIR-14 OpenLiveQ-2 Task

  2019influential citation
• Revisiting Online Personal Search Metrics with the User in Mind

  2019cites this paper
• Graded Relevance Assessments and Graded Relevance Measures of NTCIR: A Survey of the First Twenty Years

  2019influential citation
• Statistical Significance Testing in Information Retrieval: An Empirical Analysis of Type I, Type II and Type III Errors

  2019influential citation
• Information retrieval models for recommender systems

  2019influential citation
• On the interplay between normalisation, bias, and performance of paper impact metrics

  2019cites this paper
• How to Run an Evaluation Task - With a Primary Focus on Ad Hoc Information Retrieval

  2019cites this paper
• Low-cost, bottom-up measures for evaluating search result diversification

  2019cites this paper
• Evaluation Framework for Argument Retrieval in a Crowdsourced Setting

  2019cites this paper
• Estimating Measurement Uncertainty for Information Retrieval Effectiveness Metrics

  2018cites this paper
• Search Result Diversity Evaluation Based on Intent Hierarchies

  2018cites this paper
• How to evaluate rankings of academic entities using test data

  2018cites this paper
• Leveraging Uncertainty Analysis of Data to Evaluate User Influence Algorithms of Social Networks

  2018cites this paper
• Topic Set Size Design for Paired and Unpaired Data

  2018cites this paper
• Detecting Social Influence in Event Cascades by Comparing Discriminative Rankers

  2018cites this paper
• Finding Parallel Passages in Cultural Heritage Archives

  2018cites this paper
• Desirable Properties for Diversity and Truncated Effectiveness Metrics

  2018cites this paper
• Laboratory Experiments in Information Retrieval

  2018cites this paper
• On the robustness and discriminative power of information retrieval metrics for top-N recommendation

  2018influential citation
• Meta-Analysis for Retrieval Experiments Involving Multiple Test Collections

  2018cites this paper
• A Study of Per-Topic Variance on System Comparison

  2018cites this paper
• Topic Set Size Design Using Excel

  2018cites this paper
• Multiple Comparison Procedures

  2018cites this paper
• Modelling Randomness in Relevance Judgments and Evaluation Measures

  2018cites this paper
• Consistent Transformation of Ratio Metrics for Efficient Online Controlled Experiments

  2018cites this paper
• Discriminative Modeling of Social Influence for Prediction and Explanation in Event Cascades

  2018cites this paper
• Meta-evaluation of Online and Offline Web Search Evaluation Metrics

  2017cites this paper
• Evaluating Mobile Search with Height-Biased Gain

  2017cites this paper
• Efficient and effective retrieval using Higher-Order proximity models

  2017cites this paper
• Periodicity in User Engagement with a Search Engine and Its Application to Online Controlled Experiments

  2017cites this paper
• An in-depth study on diversity evaluation: The importance of intrinsic diversity

  2017cites this paper
• What Does Affect the Correlation Among Evaluation Measures?

  2017cites this paper
• Building Test Collections: An Interactive Guide for Students and Others Without Their Own Evaluation Conference Series

  2017cites this paper
• Incorporating User Expectations and Behavior into the Measurement of Search Effectiveness

  2017cites this paper
• The Probability that Your Hypothesis Is Correct, Credible Intervals, and Effect Sizes for IR Evaluation

  2017cites this paper
• geNov: A new metric for measuring novelty and relevancy in biomedical information retrieval

  2017cites this paper
• An Interval-Like Scale Property for I. R. Evaluation Measures

  2017cites this paper
• Reproducibility Challenges in Information Retrieval Evaluation

  2017cites this paper
• The anatomy of a search and mining system for digital humanities : Search And Mining Tools for Language Archives (SAMTLA)

  2017cites this paper
• Using the Delay in a Treatment Effect to Improve Sensitivity and Preserve Directionality of Engagement Metrics in A/B Experiments

  2017influential citation
• The Effect of Inter-Assessor Disagreement on IR System Evaluation: A Case Study with Lancers and Students

  2017cites this paper
• Evaluating Evaluation Measures with Worst-Case Confidence Interval Widths

  2017cites this paper
• But Is It Statistically Significant?: Statistical Significance in IR Research, 1995-2014

  2017cites this paper