Score standardization for inter-collection comparison of retrieval systems

No abstract is available for this paper.

• Publication year

  2008

• Venue

  Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

• Publication date

  2008-07-20

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/1390334.1390346
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• The relationship between IR effectiveness measures and user satisfaction

  2007cited by this paper
• A comparison of statistical significance tests for information retrieval evaluation

  2007influential reference
• Score Standardization for Robust Comparison of Retrieval Systems

  2007cited by this paper
• Hits hits TREC: exploring IR evaluation results with network analysis

  2007cited by this paper
• The influence of caption features on clickthrough patterns in web search

  2007cited by this paper
• How well does result relevance predict session satisfaction?

  2007cited by this paper
• Test theory for assessing IR test collections

  2007influential reference
• TREC: Experiment and evaluation in information retrieval

  2007cited by this paper
• Evaluating evaluation metrics based on the bootstrap

  2006cited by this paper
• User performance versus precision measures for simple search tasks

  2006cited by this paper
• A statistical method for system evaluation using incomplete judgments

  2006cited by this paper
• The maximum entropy method for analyzing retrieval measures

  2005influential reference
• Retrieval System Evaluation

  2005cited by this paper
• Information retrieval system evaluation: effort, sensitivity, and reliability

  2005cited by this paper
• Book Review: TREC: Experiment and Evaluation in Information Retrieval, edited by Ellen M. Voorhees and Donna K. Harman

  2005cited by this paper
• Cumulated gain-based evaluation of IR techniques

  2002cited by this paper
• How reliable are the results of large-scale information retrieval experiments?

  1998cited by this paper
• Statistical inference in retrieval effectiveness evaluation

  1997cited by this paper
• A Statistical Analysis of the TREC-3 Data

  1995cited by this paper
• Research and Development in Information Retrieval

  1982cited by this paper

• My System Is As Effective As Yours: Reproducibility, Sustainability, and More

  2025cites this paper
• MDCU: A Multi-Dimensional Cumulative Utility Metric for Information Retrieval

  2025cites this paper
• A Cost-Effective Framework to Evaluate LLM-Generated Relevance Judgements

  2025cites this paper
• On the quantitative analysis of assessment scores with implicit and explicit constraints

  2025cites this paper
• Evaluating Multi-Dimensional Cumulated Utility in Information Retrieval

  2025cites this paper
• SleepNet

  2024cites this paper
• Outil d'exploration visuelle de l'évaluation longitudinale en Recherche d'Information

  2024cites this paper
• Exploratory Visualization Tool for the Continuous Evaluation of Information Retrieval Systems

  2023cites this paper
• Fairness-based Evaluation of Conversational Search: A Pilot Study

  2023cites this paper
• Detecting Significant Differences Between Information Retrieval Systems via Generalized Linear Models

  2022cites this paper
• Addressing Different Evaluation Environments for Information Retrieval through Pivot Systems

  2021cites this paper
• On the Two-Sample Randomisation Test for IR Evaluation

  2021cites this paper
• Towards Meaningful Statements in IR Evaluation: Mapping Evaluation Measures to Interval Scales

  2021cites this paper
• Towards the Evaluation of Information Retrieval Systems on Evolving Datasets with Pivot Systems

  2021cites this paper
• On Fuhr's guideline for IR evaluation

  2020cites this paper
• Information Retrieval Technology: 15th Asia Information Retrieval Societies Conference, AIRS 2019, Hong Kong, China, November 7–9, 2019, Proceedings

  2020cites this paper
• Offline evaluation options for recommender systems

  2020cites this paper
• Improving the Accuracy of System Performance Estimation by Using Shards

  2019cites this paper
• A New Perspective on Score Standardization

  2019influential citation
• On the Pluses and Minuses of Risk

  2019cites this paper
• Evaluierung von Rankingverfahren für bibliothekarische Informationssysteme

  2019cites this paper
• Meta-Analysis for Retrieval Experiments Involving Multiple Test Collections

  2018influential citation
• Computing Maximized Effectiveness Distance for Recall-Based Metrics

  2018cites this paper
• Stochastic Simulation of Test Collections: Evaluation Scores

  2018cites this paper
• A Study of Per-Topic Variance on System Comparison

  2018cites this paper
• Towards Automatic Evaluation of Customer-Helpdesk Dialogues

  2018cites this paper
• Investigating per Topic Upper Bound for Session Search Evaluation

  2017cites this paper
• Using Replicates in Information Retrieval Evaluation

  2017cites this paper
• 3.5K runs, 5K topics, 3M assessments and 70M measures: What trends in 10 years of Adhoc-ish CLEF?

  2017cites this paper
• An Interval-Like Scale Property for I. R. Evaluation Measures

  2017cites this paper
• A General Linear Mixed Models Approach to Study System Component Effects

  2016cites this paper
• The CLEF Monolingual Grid of Points

  2016cites this paper
• The Effect of Score Standardisation on Topic Set Size Design

  2016influential citation
• Examining Additivity and Weak Baselines

  2016cites this paper
• A Simple and Effective Approach to Score Standardisation

  2016influential citation
• Experimental IR Meets Multilinguality, Multimodality, and Interaction

  2016cites this paper
• Two Sample T-tests for IR Evaluation: Student or Welch?

  2016cites this paper
• Document-based approach to improve the accuracy of pairwise comparison in evaluating information retrieval systems

  2015cites this paper
• Topic set size design

  2015influential citation
• CLEF 2000-2014: Lessons Learnt from Ad Hoc Retrieval

  2015cites this paper
• Hypothesis testing for the risk-sensitive evaluation of retrieval systems

  2014cites this paper
• Adaptive data fusion methods in information retrieval

  2014cites this paper
• CLEF 15th Birthday: What Can We Learn From Ad Hoc Retrieval?

  2014influential citation
• Bridging Between Information Retrieval and Databases

  2014cites this paper
• By the power of Grayskull: Small sample statistical power in Information Retrieval evaluation

  2014cites this paper
• Low-cost evaluation techniques for information retrieval systems: A review

  2013cites this paper
• Seven Numeric Properties of Effectiveness Metrics

  2013cites this paper
• Optimizing the construction of information retrieval test collections

  2013cites this paper
• Metrics, Statistics, Tests

  2013influential citation
• Information und Wissen: global, sozial und frei?

  2013cites this paper
• A generic approach to component-level evaluation in information retrieval

  2012cites this paper
• Context-based information analysis for the Web environment

  2012cites this paper
• Information retrieval through textual annotations

  2012cites this paper
• Baselines and Bigrams : Simple , Good Sentiment and Topic Classification Using bag of features + linear classifiers

  2012cites this paper
• Latent semantic indexing (LSI) fails for TREC collections

  2011influential citation
• Evaluation via Negativa of Chinese Word Segmentation for Information Retrieval

  2011cites this paper
• A multi-collection latent topic model for federated search

  2011cites this paper
• Confidence intervals for information retrieval evaluation

  2010cites this paper
• Measurement in information retrieval evaluation

  2010influential citation
• Click-based evidence for decaying weight distributions in search effectiveness metrics

  2010cites this paper
• The Effect of Pooling and Evaluation Depth on Metric Stability

  2010cites this paper
• A model for investigating and calibrating IR systems evaluation

  2010influential citation
• Entity ranking in Wikipedia: utilising categories, links and topic difficulty prediction

  2010cites this paper
• Test Collection Based Evaluation of Information Retrieval Systems

  2010cites this paper
• Measuring the Variability in Effectiveness of a Retrieval System

  2010cites this paper
• IR Evaluation without a Common Set of Topics

  2009cites this paper
• Building reliable test and training collections in information retrieval

  2009cites this paper
• Building a framework for the probability ranking principle by a family of expected weighted rank

  2009cites this paper
• Score Aggregation Techniques in Retrieval Experimentation

  2009cites this paper
• System scoring using partial prior information

  2009cites this paper
• A study on performance volatility in information retrieval

  2009cites this paper
• Topic set size redux

  2009cites this paper
• Modelling disagreement between judges for information retrieval system evaluation

  2009cites this paper
• Measuring system performance and topic discernment using generalized adaptive-weight mean

  2009cites this paper
• Topic Difficulty Prediction in Entity Ranking

  2009cites this paper
• Risk-Aware Information Retrieval

  2009cites this paper
• Weighted Rank Correlation in Information Retrieval Evaluation

  2009cites this paper
• Improvements that don't add up: ad-hoc retrieval results since 1998

  2009cites this paper
• Has adhoc retrieval improved since 1994?

  2009cites this paper
• Parameter Sensitivity in Rank-Biased Precision Yuye

  2008cites this paper
• Statistical power in retrieval experimentation

  2008cites this paper
• Exploring evaluation metrics: GMAP versus MAP

  2008cites this paper
• Rank-biased precision for measurement of retrieval effectiveness

  2008cites this paper
• Modelling A User Population for Designing Information Retrieval Metrics

  2008cites this paper
• Parameter Sensitivity in Rank-Biased Precision

  2008cites this paper