Seven pitfalls to avoid when running controlled experiments on the web

No abstract is available for this paper.

• Publication year

  2009

• Venue

  Knowledge Discovery and Data Mining

• Publication date

  2009-06-28

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/1557019.1557139
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Customer satisfaction: the customer experience through the customer's eyes

  2010cited by this paper
• SUPPLEMENT TO THE

  2009cited by this paper
• Customer Satisfaction: The customer experience through the customer's eyes

  2007cited by this paper
• Practical guide to controlled experiments on the web: listen to your customers not to the hippo

  2007cited by this paper
• An Introduction to the Bootstrap

  2007cited by this paper
• Statistics for Experimenters: Design, Innovation, and Discovery

  2006cited by this paper
• Statistics for experimenters : design, innovation, and discovery

  2005cited by this paper
• Web Robot Detection - Preprocessing Web Logfiles for Robot Detection

  2005cited by this paper
• New Developments in Classification and Data Analysis

  2005cited by this paper
• Evaluating implicit measures to improve web search

  2005cited by this paper
• Discovery of Web Robot Sessions Based on their Navigational Patterns

  2004cited by this paper
• Lessons and Challenges from Mining Retail E-Commerce Data

  2004cited by this paper
• Simpson's Paradox

  2004cited by this paper
• Learning to lead at Toyota.

  2004influential reference
• Stanford Encyclopedia of Philosophy: a dynamic reference work

  2003cited by this paper
• Design of Experiments Using the Taguchi Approach: 16 Steps to Product and Process Improvement

  2002cited by this paper
• Design of Experiments Using The Taguchi Approach: 16 Steps to Product and Process Improvement

  2001cited by this paper
• Design and Analysis of Experiments

  2001cited by this paper
• Inferring User Interest

  2001cited by this paper
• Statistical Design and Analysis of Experiments, with Applications to Engineering and Science

  1989cited by this paper
• Linear Statistical Inference and Its Applications.

  1975cited by this paper
• Linear Statistical Inference and Its Applications

  1966cited by this paper
• Linear statistical inference and its applications

  1965cited by this paper
• 新しい廣告 : Scientific advertising

  1951cited by this paper
• The Interpretation of Interaction in Contingency Tables

  1951cited by this paper
• The Biological Standardization of Insulin

  1940cited by this paper
• Responsible editor: R. Bayardo.

  year unknowncited by this paper

• Agile Development of Mobile Apps in Industrial Environment

  2025cites this paper
• Assessing Reliability in AI-Powered Learning Systems with A/A Tests

  2025influential citation
• Report on the 3rd Workshop on NeuroPhysiological Approaches for Interactive Information Retrieval (NeuroPhysIIR 2025) at SIGIR CHIIR 2025

  2025cites this paper
• Cost-Effective A/B Testing: Leveraging Go and Python for Efficient Experimentation in Hermes Testing Platform

  2024cites this paper
• An enhanced integrated fuzzy logic-based deep learning techniques (EIFL-DL) for the recommendation system on industrial applications

  2024cites this paper
• Characterization of continuous experimentation in software engineering: Expressions, models, and strategies

  2023cites this paper
• Evaluating Recommender Systems: Survey and Framework

  2022cites this paper
• A Literature Study to Characterize Continuous Experimentation in Software Engineering

  2022cites this paper
• A Cluster-Based Nearest Neighbor Matching Algorithm for Enhanced A/A Validation in Online Experimentation

  2022cites this paper
• Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology

  2022cites this paper
• Addressing Hidden Imperfections in Online Experimentation

  2022cites this paper
• A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments

  2022cites this paper
• Field Experiments

  2021cites this paper
• PlanAlyzer

  2021cites this paper
• Controlled Experimentation in Continuous Experimentation: Knowledge and Challenges

  2021cites this paper
• Website Design and Trust Elements: A/B Testing on a Start-up's Website

  2021cites this paper
• Data-Driven Approaches to User Interface Design: A Case Study

  2021cites this paper
• The HURRIER process for experimentation in business‐to‐business mission‐critical systems

  2021cites this paper
• Variance-Weighted Estimators to Improve Sensitivity in Online Experiments

  2020cites this paper
• Towards data-driven culture in a Spanish automobile manufacturer: A case study

  2020cites this paper
• Evaluating Personalization: The AB Testing Pitfalls Companies Might Not Be Aware of—A Spotlight on the Automotive Sector Websites

  2020influential citation
• Experimentation for Business-to-Business Mission-Critical Systems: A Case Study

  2020cites this paper
• Lumos: A Library for Diagnosing Metric Regressions in Web-Scale Applications

  2020cites this paper
• Exploring Agile Mobile App Development in Industrial Contexts: A Qualitative Study

  2019cites this paper
• How A/B Tests Could Go Wrong: Automatic Diagnosis of Invalid Online Experiments

  2019cites this paper
• Human-Computer Interaction: 6th Iberomarican Workshop, HCI-Collab 2020, Arequipa, Peru, September 16–18, 2020, Proceedings

  2019cites this paper
• Online Controlled Experiments at Large Scale in Society 5.0

  2019cites this paper
• Validating and Supporting a Continuous Ex- perimentation Development Environment

  2019influential citation
• ROI Calculation on Online Controlled Experiment

  2019cites this paper
• PlanAlyzer: assessing threats to the validity of online experiments

  2019cites this paper
• Effective Online Evaluation for Web Search

  2019cites this paper
• Experimentation Pitfalls to Avoid in A/B Testing for Online Personalization

  2019influential citation
• Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale

  2019cites this paper
• Multi-armed bandits in the wild: Pitfalls and strategies in online experiments

  2019cites this paper
• Top Challenges from the first Practical Online Controlled Experiments Summit

  2019cites this paper
• Citizen Science: An Information Quality Research Frontier

  2019cites this paper
• Business process improvement with the AB-BPM methodology

  2019cites this paper
• Testing Predictive Models in Production

  2018cites this paper
• Online Controlled Experiments for Personalised e-Commerce Strategies: Design, Challenges, and Pitfalls

  2018cites this paper
• Consistent Transformation of Ratio Metrics for Efficient Online Controlled Experiments

  2018influential citation
• Continuous Experimentation and A/B Testing: A Mapping Study

  2018cites this paper
• Winner's Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments

  2018cites this paper
• Search-Based Scheduling of Experiments in Continuous Deployment

  2018cites this paper
• Online Evaluation for Effective Web Service Development

  2018cites this paper
• Shadow Testing for Business Process Improvement

  2018cites this paper
• Effective Online Controlled Experiment Analysis at Large Scale

  2018cites this paper
• An Activity and Metric Model for Online Controlled Experiments

  2018cites this paper
• Experimentation growth: Evolving trustworthy A/B testing capabilities in online software companies

  2018cites this paper
• Towards Automated Experiments in Software Intensive Systems

  2018influential citation
• Safely and Quickly Deploying New Features with a Staged Rollout Framework Using Sequential Test and Adaptive Experimental Design

  2018cites this paper
• Semantically enhanced cross-domain recommender systems

  2018cites this paper
• Online Evaluation of Rankers Using Multileaving

  2018cites this paper
• Online Controlled Experiments for Personalised e-Commerce Strategies

  2018cites this paper
• Beyond Success Rate: Utility as a Search Quality Metric for Online Experiments

  2017cites this paper
• Characterizing Experimentation in Continuous Deployment: A Case Study on Bing

  2017cites this paper
• Using the Delay in a Treatment Effect to Improve Sensitivity and Preserve Directionality of Engagement Metrics in A/B Experiments

  2017cites this paper
• Leaky Abstraction In Online Experimentation Platforms: A Conceptual Framework To Categorize Common Challenges

  2017cites this paper
• Introduction: Second Edition

  2017cites this paper
• Periodicity in User Engagement with a Search Engine and Its Application to Online Controlled Experiments

  2017influential citation
• The Benefits of Controlled Experimentation at Scale

  2017cites this paper
• Faster online experimentation by eliminating traditional A/A validation

  2017cites this paper
• AB-BPM: Performance-Driven Instance Routing for Business Process Improvement

  2017cites this paper
• Inform Product Change through Experimentation with Data-Driven Behavioral Segmentation

  2017cites this paper
• Online and ubiquitous HCI research

  2017cites this paper
• Issues with conducting controlled on-line experiments for E-Commerce

  2017cites this paper
• Towards a Data-Driven Pricing Decision

  2016cites this paper
• Online Experimentation Diagnosis and Troubleshooting Beyond AA Validation

  2016cites this paper
• iMTracker: um programa para monitoramento remoto das ações de usuários de sistemas Web

  2016cites this paper
• A Nonparametric Sequential Test for Online Randomized Experiments

  2016cites this paper
• Interface Design Optimization as a Multi-Armed Bandit Problem

  2016cites this paper
• Boosted Decision Tree Regression Adjustment for Variance Reduction in Online Controlled Experiments

  2016cites this paper
• Weighted Gaussian Process for Estimating Treatment Effect

  2016cites this paper
• Extreme States Distribution Decomposition Method for Search Engine Online Evaluation

  2015influential citation
• Gamifying online marketplaces to overcome supply and demand imbalances: A design and evaluation study at OLX India

  2015cites this paper
• Exploring the Possibilities of Split Testing on Motion-Based Games

  2015cites this paper
• Looking at Everything in Context

  2015cites this paper
• Confidence intervals for AB-test

  2015influential citation
• Website Optimization Problem and Its Solutions

  2015cites this paper
• From Infrastructure to Culture: A/B Testing Challenges in Large Scale Social Networks

  2015cites this paper
• Online Controlled Experiments and A/B Tests

  2015cites this paper
• A Short Survey on Online and Offline Methods for Search Quality Evaluation

  2015cites this paper
• Sign-Aware Periodicity Metrics of User Engagement for Online Search Quality Evaluation

  2015cites this paper
• Practical Aspects of Sensitivity in Online Experimentation with User Engagement Metrics

  2015influential citation
• Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments

  2015influential citation
• Understanding User Behavior Through Log Data and Analysis

  2014cites this paper
• Seven rules of thumb for web site experimenters

  2014influential citation
• Why marketplace experimentation is harder than it seems: the role of test-control interference

  2014cites this paper
• Towards Automated A/B Testing

  2014cites this paper
• Continuous Experimentation in the B2B Domain: A Case Study

  2014cites this paper
• Designing and deploying online field experiments

  2014cites this paper
• Do you know the speaker?: an online experiment with authority messages on event websites

  2014cites this paper
• Webpage Optimization Problem and Its Optimization Method

  2014cites this paper
• 2 3 D ec 2 01 3 Predictive User Modeling with Actionable Attributes Technical Report

  2013cites this paper
• Predictive User Modeling with Actionable Attributes

  2013cites this paper
• Uncertainty in online experiments with dependent data: an evaluation of bootstrap methods

  2013cites this paper
• Der Nutzen informetrischer Analysen und nicht-textueller Dokumentattribute für das Information Retrieval in digitalen Bibliotheken

  2013cites this paper
• Online controlled experiments at large scale

  2013cites this paper
• Information Retrieval und Informetrie: Zur Anwendung informetrischer Methoden in digitalen Bibliotheken

  2013cites this paper
• Leveraging User Testing to Address Learnability Issues for Teachers Using ASSISTments

  2013cites this paper
• Case study on the business value impact of personalized recommendations on a large online retailer

  2012influential citation