Test & Roll: Profit-Maximizing A/B Tests

We reframe A/B testing as a decision problem with the goal of maximizing profit and derive the optimal sample size formula.

• Publication year

  2018

• Venue

  Marketing science (Providence, R.I.)

• Publication date

  2018-11-01

• Fields of study

  Mathematics, Business, Economics, Computer Science

• Identifiers
  DOI 10.2139/ssrn.3274875arXiv 1811.00457
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• A/B Testing

  2021cited by this paper
• A/B Testing with Fat Tails

  2020cited by this paper
• Dynamic Online Pricing with Incomplete Information Using Multiarmed Bandit Experiments

  2019cited by this paper
• Efficiently Evaluating Targeting Policies: Improving Upon Champion vs. Challenger Experiments

  2019cited by this paper
• Heterogeneous Treatment Effects and Optimal Targeting Policy Evaluation

  2018cited by this paper
• p-Hacking and False Discovery in A/B Testing

  2018cited by this paper
• Evaluating and Improving Targeting Policies with Field Experiments Using Counterfactual Policy Logging

  2018cited by this paper
• 3-2009 Economic Analysis of Simulation Selection Problems

  2017cited by this paper
• Measuring Multi-Channel Advertising Response

  2017cited by this paper
• A Decision Theoretic Approach to A/B Testing

  2017cited by this paper
• Customer Acquisition via Display Advertising Using Multi-Armed Bandit Experiments

  2017influential reference
• Determination of the optimal sample size for a clinical trial accounting for the population size

  2016cited by this paper
• Measuring Multi-Channel Advertising Effectiveness Using Consumer-Level Advertising Response Data

  2016cited by this paper
• The New Stats Engine

  2015cited by this paper
• The Unfavorable Economics of Measuring the Returns to Advertising

  2015influential reference
• Which Products Are Best Suited to Mobile Advertising? A Field Study of Mobile Display Advertising Effects on Consumer Attitudes and Intentions

  2014cited by this paper
• Further Optimal Regret Bounds for Thompson Sampling

  2012cited by this paper
• Sequential Sampling with Economics of Selection Procedures

  2012cited by this paper
• A modern Bayesian look at the multi-armed bandit

  2010cited by this paper
• Approximate sample size formulas for the two-sample trimmed mean test with unequal variances.

  2007cited by this paper
• A Learning Approach for Interactive Marketing to a Customer Segment

  2007cited by this paper
• Real-Time Evaluation of E-mail Campaign Performance

  2006cited by this paper
• Choosing sample size for a clinical trial using decision analysis

  2003influential reference
• Improving Parameter Estimates and Model Prediction by Aggregate Customization in Choice Experiments

  2001cited by this paper
• New Two-Stage and Sequential Procedures for Selecting the Best Simulated System

  2001cited by this paper
• Mailing Decisions in the Catalog Sales Industry

  1996cited by this paper
• Decision making during a phase III randomized controlled trial.

  1994cited by this paper
• Optimal Statistical Decisions

  1970cited by this paper
• ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLES

  1933influential reference

• Large Language Models and Creative Content Design: a case study of email marketing at Wine Access

  2026cites this paper
• Getting the Most Out of A/B Tests Using the Asymptotic Minimax-Regret Criteria

  2025cites this paper
• Optimal pricing and promotional efforts strategies in a competitive live-streaming supply chain

  2025cites this paper
• Theory Instead of Experiment (TIE): A Creator Valuation System at Tencent

  2025cites this paper
• Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspective

  2025cites this paper
• Observational vs. Experimental Data When Making Automated Decisions Using Machine Learning

  2025cites this paper
• A Sample Size Calculation for Training and Certifying Targeting Policies

  2025cites this paper
• Heterogeneous treatment effects and optimal targeting policy evaluation

  2024cites this paper
• Reducing Interference Bias in Online Marketplace Experiments Using Cluster Randomization: Evidence from a Pricing Meta-experiment on Airbnb

  2024cites this paper
• The Role of Artificial Intelligence in Integrated Marketing Communication: An Evaluation of ChatGPT

  2024cites this paper
• Optimizing Returns from Experimentation Programs

  2024cites this paper
• When Is Heterogeneity Actionable for Personalization?

  2024cites this paper
• Embracing entrepreneurship in the creator economy: The rise of creatrepreneurs

  2024cites this paper
• Ranking by Lifts: A Cost-Benefit Approach to Large-Scale A/B Tests

  2024cites this paper
• Probabilistic Machine Learning: New Frontiers for Modeling Consumers and their Choices

  2024cites this paper
• Optimal Prototyping on Experimentation Platforms

  2023cites this paper
• Optimal Design of Controlled Experiments for Personalized Decision Making in the Presence of Observational Covariates

  2023cites this paper
• Frontiers: Which Firms Gain from Digital Advertising? Evidence from a Field Experiment

  2023cites this paper
• Unlimited Testing: Let's Test Your Emails with AI

  2023cites this paper
• A/B Testing: A Systematic Literature Review

  2023cites this paper
• Business Policy Experiments using Fractional Factorial Designs: Consumer Retention on DoorDash

  2023cites this paper
• How to Maximize Clicks for Display Advertisement in Digital Marketing? A Reinforcement Learning Approach

  2022cites this paper
• Editorial: Marketing's Role in the Evolving Discipline of Product Management

  2022cites this paper
• A 680,000-person megastudy of nudges to encourage vaccination in pharmacies

  2022cites this paper
• Causal Decision Making and Causal Effect Estimation Are Not the Same... and Why It Matters

  2021cites this paper
• Profit-Driven Experimental Design

  2021cites this paper
• False Discovery in A/B Testing

  2021cites this paper
• Theory development in servitization through the application of fsQCA and experiments

  2021cites this paper
• A Mega-Study of Text-Based Nudges Encouraging Patients to Get Vaccinated at an Upcoming Doctor’s Appointment

  2021cites this paper
• False Discovery in A/B Testing

  2020cites this paper
• Combining Observational and Experimental Data to Improve Efficiency Using Imperfect Instruments

  2020cites this paper
• A/B Testing with Fat Tails

  2020cites this paper
• Capturing Marketing Information to Fuel Growth

  2020cites this paper
• Combining Observational and Experimental Data Using First-stage Covariates

  2020influential citation
• References

  2020cites this paper
• Latent Stratification for Incrementality Experiments

  2019cites this paper
• Inefficiencies in Digital Advertising Markets

  2019cites this paper
• Design and Evaluation of Personalized Targeting Policies: Application to Free Trials

  2019cites this paper
• Principal Stratification for Advertising Experiments∗

  2019cites this paper
• Online Causal Inference for Advertising in Real-Time Bidding Auctions

  2019influential citation
• Empirical Bayes Estimation of Treatment Effects with Many A/B Tests: An Overview

  2019cites this paper
• Principal Stratification for Advertising Experiments

  2019cites this paper
• Informed Bayesian Inference for the A/B Test

  2019cites this paper
• Online Inference for Advertising Auctions

  2019cites this paper