Robust inference for the treatment effect variance in experiments using machine learning

Experimenters often collect baseline data to study heterogeneity. I propose the first valid confidence intervals for the VCATE, the treatment effect variance explained by observables. Conventional approaches yield incorrect coverage when the VCATE is zero. As a result, practitioners could be prone to detect heterogeneity even when none exists. The reason why coverage worsens at the boundary is that all efficient estimators have a locally-degenerate influence function and may not be asymptotically normal. I solve the problem for a broad class of multistep estimators with a predictive first stage. My confidence intervals account for higher-order terms in the limiting distribution and are fast to compute. I also find new connections between the VCATE and the problem of deciding whom to treat. The gains of targeting treatment are (sharply) bounded by half the square root of the VCATE. Finally, I document excellent performance in simulation and reanalyze an experiment from Malawi.

• Publication year

  2023

• Venue

  Unknown venue

• Publication date

  2023-06-06

• Fields of study

  Computer Science, Economics

• Identifiers
  arXiv 2306.03363
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Cluster-robust inference: A guide to empirical practice

  2022cited by this paper
• Machine Learning for Variance Reduction in Online Experiments

  2021cited by this paper
• Fair Policy Targeting

  2020cited by this paper
• Generic results for establishing the asymptotic size of confidence sets and tests

  2020influential reference
• Methods to integrate multinormals and compute classification measures

  2020cited by this paper
• The Mortality and Medical Costs of Air Pollution: Evidence from Changes in Wind Direction.

  2019cited by this paper
• Parents’ Beliefs about Their Children’s Academic Ability: Implications for Educational Investments

  2019influential reference
• Automatic Debiased Machine Learning of Causal and Structural Effects

  2018influential reference
• A fundamental measure of treatment effect heterogeneity

  2018influential reference
• Rethinking the Benefits of Youth Employment Programs: The Heterogeneous Effects of Summer Jobs

  2017cited by this paper
• Double/debiased machine learning for treatment and structural parameters

  2017influential reference
• Generic Machine Learning Inference on Heterogenous Treatment Effects in Randomized Experiments

  2017influential reference
• Policy Learning With Observational Data

  2017influential reference
• Locally Robust Semiparametric Estimation

  2016influential reference
• Decomposing Treatment Effect Variation

  2016influential reference
• Model Selection for Treatment Choice: Penalized Welfare Maximization

  2016cited by this paper
• Generalized random forests

  2016cited by this paper
• Who should be Treated? Empirical Welfare Maximization Methods for Treatment Choice

  2015influential reference
• NBER WORKING PAPER SERIES CAN VARIATION IN SUBGROUPS' AVERAGE TREATMENT EFFECTS EXPLAIN TREATMENT EFFECT HETEROGENEITY? EVIDENCE FROM A SOCIAL EXPERIMENT

  2014cited by this paper
• Program evaluation and causal inference with high-dimensional data

  2013cited by this paper
• Supplementary Appendix for"Inference on Treatment Effects After Selection Amongst High-Dimensional Controls"

  2013cited by this paper
• SIMULTANEOUS ANALYSIS OF LASSO AND DANTZIG SELECTOR

  2008cited by this paper
• Nonparametric Tests for Treatment Effect Heterogeneity

  2008influential reference
• Statistical treatment rules for heterogeneous populations

  2003influential reference
• Asymptotics for lasso-type estimators

  2000cited by this paper
• Improved Rates and Asymptotic Normality for Nonparametric Neural Network Estimators

  1999cited by this paper
• Elements of large-sample theory

  1998influential reference
• Making the Most out of Programme Evaluations and Social Experiments: Accounting for Heterogeneity in Programme Impacts

  1997cited by this paper
• Semiparametric Efficiency Bounds

  1990influential reference

• Estimating Treatment Effects Under Bounded Heterogeneity

  2025cites this paper
• Nonparametric inference on non-negative dissimilarity measures at the boundary of the parameter space

  2023cites this paper