Adaptive Sample-Level Framework Motivated by Distributionally Robust Optimization with Variance-Based Radius Assignment for Enhanced Neural Network Generalization Under Distribution Shift

Distribution shifts and minority subpopulations frequently undermine the reliability of deep neural networks trained using Empirical Risk Minimization (ERM). Distributionally Robust Optimization (DRO) addresses this by optimizing for the worst-case risk within a neighborhood of the training distribution. However, conventional methods depend on a single, global robustness budget, which can lead to overly conservative models or a misallocation of robustness. We propose a variance-driven, adaptive, sample-level DRO (Var-DRO) framework that automatically identifies high-risk training samples and assigns a personalized robustness budget to each based on its online loss variance. Our formulation employs two-sided, KL-divergence-style bounds to constrain the ratio between adversarial and empirical weights for every sample. This results in a linear inner maximization problem over a convex polytope, which admits an efficient water-filling solution. To stabilize training, we introduce a warmup phase and a linear ramp schedule for the global cap on per-sample budgets, complemented by label smoothing for numerical robustness. Evaluated on CIFAR-10-C (corruptions), our method achieves the highest overall mean accuracy compared to ERM and KL-DRO. On Waterbirds, Var-DRO improves overall performance while matching or surpassing KL-DRO. On the original CIFAR-10 dataset, Var-DRO remains competitive, exhibiting the modest trade-off anticipated when prioritizing robustness. The proposed framework is unsupervised (requiring no group labels), straightforward to implement, theoretically sound, and computationally efficient.

• Publication year

  2025

• Venue

  arXiv.org

• Publication date

  2025-11-04

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.48550/arXiv.2511.05568arXiv 2511.05568
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Reformulation and Enhancement of Distributed Robust Optimization Framework Incorporating Decision-Adaptive Uncertainty Sets

  2024cited by this paper
• Robust Distribution Learning with Local and Global Adversarial Corruptions

  2024cited by this paper
• Modeling the Second Player in Distributionally Robust Optimization

  2021cited by this paper
• Distributionally Robust Neural Networks

  2020cited by this paper
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

  2019cited by this paper
• Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach

  2016cited by this paper
• Stochastic Gradient Methods for Distributionally Robust Optimization with f-divergences

  2016cited by this paper
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

  2016cited by this paper
• Robust Solutions of Optimization Problems Affected by Uncertain Probabilities

  2013cited by this paper

• No citing papers are available for this paper.