Evaluating the crowd with confidence

Worker quality control is a crucial aspect of crowdsourcing systems; typically occupying a large fraction of the time and money invested on crowdsourcing. In this work, we devise techniques to generate confidence intervals for worker error rate estimates, thereby enabling a better evaluation of worker quality. We show that our techniques generate correct confidence intervals on a range of real-world datasets, and demonstrate wide applicability by using them to evict poorly performing workers, and provide confidence intervals on the accuracy of the answers.

• Publication year

  2013

• Venue

  Knowledge Discovery and Data Mining

• Publication date

  2013-08-11

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/2487575.2487595arXiv 1411.6562
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Smart Drill-Down: A New Data Exploration Operator

  2014cited by this paper
• Reflections on Stanford's MOOCs

  2013cited by this paper
• CrowdScreen: algorithms for filtering data with humans

  2012cited by this paper
• Active sampling for entity matching

  2012cited by this paper
• Statistical Inference

  2012cited by this paper
• Learning from crowds in the presence of schools of thought

  2012cited by this paper
• CrowdER: Crowdsourcing Entity Resolution

  2012cited by this paper
• CDAS: A Crowdsourcing Data Analytics System

  2012cited by this paper
• Eliminating Spammers and Ranking Annotators for Crowdsourced Labeling Tasks

  2012cited by this paper
• Identifying Reliable Workers Swiftly

  2012cited by this paper
• So who won?: dynamic max discovery with the crowd

  2012cited by this paper
• Crowdclustering

  2011cited by this paper
• Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems

  2011cited by this paper
• Crowdsourcing subjective image quality evaluation

  2011cited by this paper
• Human-powered Sorts and Joins

  2011cited by this paper
• Human-assisted graph search: it's okay to ask questions

  2011cited by this paper
• Theory and Use of the EM Algorithm

  2011cited by this paper
• crowdsourcing : rating annotators and obtaining cost-effective labels

  2010cited by this paper
• Re: CAPTCHAs-Understanding CAPTCHA-Solving Services in an Economic Context

  2010cited by this paper
• Feasibility of Human-in-the-loop Minimum Error Rate Training

  2009cited by this paper
• Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise

  2009cited by this paper
• Supervised learning from multiple experts: whom to trust when everyone lies a bit

  2009cited by this paper
• Efficiently learning the accuracy of labeling sources for selective sampling

  2009influential reference
• Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

  2008cited by this paper
• Crowdsourcing for relevance evaluation

  2008cited by this paper
• Get another label? improving data quality and data mining using multiple, noisy labelers

  2008cited by this paper
• All of Statistics

  2004influential reference
• Dealing with missing data: Part II

  2001cited by this paper
• The EM algorithm and extensions

  1996cited by this paper
• Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm

  1979influential reference
• Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper

  1977influential reference
• Probable Inference, the Law of Succession, and Statistical Inference

  1927cited by this paper

• Scaling Success: A Systematic Review of Peer Grading Strategies for Accuracy, Efficiency, and Learning in Contemporary Education

  2025cites this paper
• Reliability and Stability of Mean Opinion Score for Image Aesthetic Quality Assessment Obtained Through Crowdsourcing

  2024cites this paper
• Combining Human and Machine Confidence in Truthfulness Assessment

  2022cites this paper
• Human-in-the-Loop Systems for Truthfulness: A Study of Human and Machine Confidence

  2021cites this paper
• Comparing the Value of Labeled and Unlabeled Data in Method-of-Moments Latent Variable Estimation

  2021cites this paper
• Empirical Study on Effects of Self-Correction in Crowdsourced Image Classification Tasks

  2021cites this paper
• Strong natural language query generation

  2021cites this paper
• Toward Annotator Group Bias in Crowdsourcing

  2021cites this paper
• Universalizing Weak Supervision

  2021cites this paper
• Proceedings of the 2021 Truth and Trust Online Conference (TTO 2021), Virtual, October 7-8, 2021

  2021cites this paper
• Special Issue on Human Powered AI Systems

  2020cites this paper
• Evaluating the crowd quality for subjective questions based on a Spark computing environment

  2020cites this paper
• Trends and Applications in Knowledge Discovery and Data Mining: PAKDD 2020 Workshops, DSFN, GII, BDM, LDRC and LBD, Singapore, May 11–14, 2020, Revised Selected Papers

  2020cites this paper
• Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods

  2020cites this paper
• Human-in-the-loop Techniques in Machine Learning

  2020cites this paper
• Towards Expert Preference on Academic Article Recommendation Using Bibliometric Networks

  2020cites this paper
• A Conceptual Professional Assessment Model Based RDF Data Crowdsourcing

  2020cites this paper
• Knowledge Enhanced Quality Estimation for Crowdsourcing

  2019cites this paper
• Linked Data Crowdsourcing Quality Assessment based on Domain Professionalism

  2019cites this paper
• Capturing Human Factors to Optimize Crowdsourced Label Acquisition through Active Learning

  2019cites this paper
• Crowd Sourcing-based Deduplication in Big Data Environment

  2019cites this paper
• The Ever Evolving Online Labor Market: Overview, Challenges and Opportunities

  2019cites this paper
• Quality Management of Workers in an In-House Crowdsourcing-Based Framework for Deduplication of Organizations’ Databases

  2019cites this paper
• Crowd-Powered Data Mining

  2018cites this paper
• Millionaire: a hint-guided approach for crowdsourcing

  2018cites this paper
• CrowdSem 2013 : Crowdsourcing the Semantic Web

  2018cites this paper
• Pay-per-Question: Towards Targeted Q&A with Payments

  2018cites this paper
• Spatial Keyword Querying: Ranking Evaluation and Efficient Query Processing

  2018cites this paper
• A partial-order-based framework for cost-effective crowdsourced entity resolution

  2018cites this paper
• Further Insights on Drawing Sound Conclusions from Noisy Judgments

  2018cites this paper
• Human Factors Modeling in Crowdsourcing

  2018cites this paper
• Quality Control in Crowdsourcing

  2018cites this paper
• Crowdsourced promotions in doubt: Analyzing effective crowdsourced promotions

  2018cites this paper
• An Empirical Study on Short- and Long-Term Effects of Self-Correction in Crowdsourced Microtasks

  2018cites this paper
• Analyzing Payment-Driven Targeted Q8A Systems

  2018cites this paper
• Towards Monetary Incentives in Social Q&A Services

  2017cites this paper
• Empirical Study on Assessment Algorithms with Confidence in Crowdsourcing

  2017cites this paper
• CDB: Optimizing Queries with Crowd-Based Selections and Joins

  2017cites this paper
• Crowdsourced Data Management: Overview and Challenges

  2017cites this paper
• An online statistical quality control framework for performance management in crowdsourcing

  2017influential citation
• Detecting Low-Quality Workers in QoE Crowdtesting: A Worker Behavior-Based Approach

  2017cites this paper
• A crowdsourced “Who wants to be a millionaire?” player

  2017cites this paper
• Using hybrid algorithmic-crowdsourcing methods for academic knowledge acquisition

  2017cites this paper
• Probabilistic Entity Resolution with Imperfect Crowd

  2017cites this paper
• Drawing Sound Conclusions from Noisy Judgments

  2017cites this paper
• Leveraging crowd skills and consensus for collaborative web-resource labeling

  2017cites this paper
• Crowdsourcing for data management

  2017cites this paper
• Truth Inference in Crowdsourcing: Is the Problem Solved?

  2017cites this paper
• CrowdDQS: Dynamic Question Selection in Crowdsourcing Systems

  2017cites this paper
• Managing the quality of crowdsourced databases

  2017cites this paper
• Crowdsourcing Based Evaluation of Ranking Approaches for Spatial Keyword Querying

  2017cites this paper
• Select Your Questions Wisely: For Entity Resolution With Crowd Errors

  2017cites this paper
• Message From the Incoming Editor-in-Chief

  2017cites this paper
• Data Integration with Dynamic Data Sources

  2016cites this paper
• PANDA: A platform for academic knowledge discovery and acquisition

  2016cites this paper
• Optimizing Open-Ended Crowdsourcing: The Next Frontier in Crowdsourced Data Management

  2016cites this paper
• Energy big data: A survey

  2016cites this paper
• Quality Assurance Strategies in Microtask Crowdsourcing

  2016cites this paper
• Confusions over Time: An Interpretable Bayesian Model to Characterize Trends in Decision Making

  2016cites this paper
• Crowdsourced Data Management: A Survey

  2016influential citation
• Lightly Supervised Quality Estimation

  2016cites this paper
• Toward Worker-Centric Crowdsourcing

  2016cites this paper
• Towards Globally Optimal Crowdsourcing Quality Management: The Uniform Worker Setting

  2016influential citation
• A computational model for refining Data domains in the property reconciliation

  2016cites this paper
• A Crowdsourcing Worker Quality Evaluation Algorithm on MapReduce for Big Data Applications

  2016cites this paper
• Combining crowd consensus and user trustworthiness for managing collective tasks

  2016cites this paper
• Crowdsourcing Reliable Ratings for Underexposed Items

  2016cites this paper
• Pay-as-you-go instance-level integration

  2016cites this paper
• Concordance in the Crowdsourcing Activity

  2016cites this paper
• Empirical Methods in Peer Prediction

  2016cites this paper
• CONSENSUS-BASED CROWDSOURCING: TECHNIQUES AND APPLICATIONS

  2015cites this paper
• Acquiring Reliable Ratings from the Crowd

  2015cites this paper
• Effective Recommendations for Leisure Activities

  2015cites this paper
• Cognition and Statistical-Based Crowd Evaluation Framework for ER-in-House Crowdsourcing System: Inbound Contact Center

  2015cites this paper
• История, направления и некоторые проблемы современных исследований краудсорсинга как научно-практической дисциплины

  2015cites this paper
• Motivating students with new mechanisms of online assignments and examination to meet the MOOC challenges for programming

  2015cites this paper
• Databases Theory and Applications

  2015cites this paper
• Crowdsourced Data Management: Industry and Academic Perspectives

  2015cites this paper
• Understanding Student Motivation, Behaviors and Perceptions in MOOCs

  2015cites this paper
• Exploiting Features for Data Source Quality Estimation

  2015cites this paper
• Fault-Tolerant Entity Resolution with the Crowd

  2015cites this paper
• Measuring Crowdsourcing Effort with Error-Time Curves

  2015cites this paper
• Worker Skill Estimation in Team-Based Tasks

  2015cites this paper
• Globally Optimal Crowdsourcing Quality Management

  2015cites this paper
• Sensor Selection for Crowdsensing Dynamical Systems

  2015cites this paper
• A Bayesian Framework for Modeling Human Evaluations

  2015cites this paper
• SLiMFast: Guaranteed Results for Data Fusion and Source Reliability

  2015cites this paper
• Cheaper and Better: Selecting Good Workers for Crowdsourcing

  2015cites this paper
• Finish Them!: Pricing Algorithms for Human Computation

  2014cites this paper
• Optimal Crowd-Powered Rating and Filtering Algorithms

  2014cites this paper
• Crowdsourcing of Formally-Specified Computations

  2014cites this paper
• Comprehensive and reliable crowd assessment algorithms

  2014cites this paper
• Collusion-Resistant Evaluation of Side-Choosing Games

  2014cites this paper
• A Weighted Aggregation Rule in Crowdsourcing Systems for High Result Accuracy

  2014cites this paper
• FSCP : A Platform for Crowdsourcing Formal Science

  2013cites this paper
• Towards a generic framework for trustworthy spatial crowdsourcing

  2013cites this paper
• Domain-Independent Quality Measures for Crowd Truth Disagreement

  2013cites this paper
• What makes a great MOOC? An interdisciplinary analysis of student retention in online courses

  2013cites this paper
• Crowdsourced Semantics with Semantic Tagging: "Don't just tag it, LexiTag it!"

  2013influential citation
• Crowdsourcing of Computational Problem Solving

  2013cites this paper