NEXT: A system to easily connect crowdsourcing and adaptive data collection

Obtaining useful crowdsourcing results often requires more responses than can be easily collected. Reducing the number of responses required can be done by adapting to previous responses with "adaptive" sampling algorithms, but these algorithms present a fundamental challenge when paired with crowdsourcing. At UW–Madison, we have built a powerful crowdsourcing data collection tool called NEXT (http://nextml.org) that can be used with arbitrary adaptive algorithms. Each week, our system is used by The New Yorker to run their Cartoon Caption contest (http://www.newyorker.com/cartoons/vote). In this paper, we will explain what NEXT is and it’s applications, architecture and experimentalist use.

• Publication year

  2017

• Venue

  SciPy

• Publication date

  Unknown publication date

• Fields of study

  Computer Science

• Identifiers
  DOI 10.25080/SHINMA-7F4C6E7-010
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Best arm identification in multi-armed bandits with delayed feedback

  2018cited by this paper
• Anytime Exploration for Multi-armed Bandits using Confidence Information

  2016cited by this paper
• How to Model Implicit Knowledge? Similarity Learning Methods to Assess Perceptions of Visual Representations

  2016cited by this paper
• A Multiworld Testing Decision Service

  2016cited by this paper
• Finite Sample Prediction and Recovery Bounds for Ordinal Embedding

  2016cited by this paper
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

  2015cited by this paper
• NEXT: A System for Real-World Development, Evaluation, and Application of Active Learning

  2015cited by this paper
• Going deeper with convolutions

  2014cited by this paper
• Very Deep Convolutional Networks for Large-Scale Image Recognition

  2014cited by this paper
• Information Complexity in Bandit Subset Selection

  2013cited by this paper
• Pairwise ranking aggregation in a crowdsourced setting

  2013cited by this paper
• lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits

  2013cited by this paper
• Best Arm Identification: A Unified Approach to Fixed Budget and Fixed Confidence

  2012cited by this paper
• The K-armed Dueling Bandits Problem

  2012cited by this paper
• ImageNet: A large-scale hierarchical image database

  2009cited by this paper
• Supervised Machine Learning: A Review of Classification Techniques

  2007cited by this paper
• A General Agnostic Active Learning Algorithm

  2007cited by this paper
• Faster Rates in Regression via Active Learning

  2005cited by this paper
• Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence, by John H. Holland MIT Press (Bradford Books), Cambridge, Mass., 1992, xiv+211 pp. (Paperback £13.50, cloth £26.95)

  1993cited by this paper
• Quicksort

  1962cited by this paper

• The Dimensions of dimensionality.

  2024cites this paper
• Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning

  2024cites this paper
• A classification and review of tools for developing and interacting with machine learning systems

  2022cites this paper
• Semantic Similarity of Social Functional Smiles and Laughter

  2022cites this paper
• Evidence for Distinct Facial Signals of Reward, Affiliation, and Dominance from Both Perception and Production Tasks

  2021cites this paper
• Enriching ImageNet with Human Similarity Judgments and Psychological Embeddings

  2020cites this paper
• Popular Imperceptibility Measures in Visual Adversarial Attacks are Far from Human Perception

  2020cites this paper
• A robot and comedian walk into a bar, and. . . AHA!

  2019cites this paper
• Should Adversarial Attacks Use Pixel p-Norm?

  2019cites this paper
• Obtaining psychological embeddings through joint kernel and metric learning

  2019cites this paper
• SMART: An Open Source Data Labeling Platform for Supervised Learning

  2018cites this paper
• Can Artificial Intelligence Help in Ideation? A Theory-Based Model for Idea Screening in Crowdsourcing Contests

  year unknowncites this paper