Multiple Source Adaptation and the Rényi Divergence

This paper presents a novel theoretical study of the general problem of multiple source adaptation using the notion of Renyi divergence. Our results build on our previous work [12], but significantly broaden the scope of that work in several directions. We extend previous multiple source loss guarantees based on distribution weighted combinations to arbitrary target distributions P, not necessarily mixtures of the source distributions, analyze both known and unknown target distribution cases, and prove a lower bound. We further extend our bounds to deal with the case where the learner receives an approximate distribution for each source instead of the exact one, and show that similar loss guarantees can be achieved depending on the divergence between the approximate and true distributions. We also analyze the case where the labeling functions of the source domains are somewhat different. Finally, we report the results of experiments with both an artificial data set and a sentiment analysis task, showing the performance benefits of the distribution weighted combinations and the quality of our bounds based on the Renyi divergence.

• Publication year

  2009

• Venue

  Conference on Uncertainty in Artificial Intelligence

• Publication date

  2009-06-18

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1205.2628
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Statistical Learning Theory

  2021cited by this paper
• Domain Adaptation with Multiple Sources

  2008influential reference
• Learning Bounds for Domain Adaptation

  2007cited by this paper
• Instance Weighting for Domain Adaptation in NLP

  2007cited by this paper
• Frustratingly Hard Domain Adaptation for Dependency Parsing

  2007cited by this paper
• Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification

  2007cited by this paper
• Domain Adaptation for Statistical Classifiers

  2006cited by this paper
• Learning from Multiple Sources

  2006cited by this paper
• Analysis of Representations for Domain Adaptation

  2006cited by this paper
• Supervised and unsupervised PCFG adaptation to novel domains

  2003cited by this paper
• Information Measures: Information and its Description in Science and Engineering

  2001influential reference
• Statistical methods for speech recognition

  1997cited by this paper
• A maximum entropy approach to adaptive statistical language modelling

  1996cited by this paper
• Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models

  1995cited by this paper
• Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains

  1994cited by this paper
• Adaptive Language Modeling Using Minimum Discriminant Estimation

  1992cited by this paper
• A theory of the learnable

  1984cited by this paper
• IEEE Transactions on Pattern Analysis and Machine Intelligence 24(6):748-763, 2002 Recognizing Imprecisely Localized, Partially Occluded and Expression Variant Faces from a Single Sample per Class

  year unknowncited by this paper

• Learning from Biased and Costly Data Sources: Minimax-optimal Data Collection under a Budget

  2026cites this paper
• A Theoretical Framework for Modular Learning of Robust Generative Models

  2026cites this paper
• On The Statistical Complexity of Offline Decision-Making

  2025cites this paper
• Synergizing Deconfounding and Temporal Generalization For Time-series Counterfactual Outcome Estimation

  2025cites this paper
• An Information-Theoretic Framework for Out-of-Distribution Generalization With Applications to Stochastic Gradient Langevin Dynamics

  2025cites this paper
• On the Hardness of Unsupervised Domain Adaptation: Optimal Learners and Information-Theoretic Perspective

  2025cites this paper
• Integral Imprecise Probability Metrics

  2025cites this paper
• Understanding and Robustifying Sub-domain Alignment for Domain Adaptation

  2025cites this paper
• Purifying Approximate Differential Privacy with Randomized Post-processing

  2025cites this paper
• Stein Discrepancy for Unsupervised Domain Adaptation

  2025cites this paper
• Transfer Learning Beyond Bounded Density Ratios

  2024cites this paper
• Gradual Fine-Tuning with Graph Routing for Multi-Source Unsupervised Domain Adaptation

  2024influential citation
• Resolving Domain Shift For Representations Of Speech In Non-Invasive Brain Recordings

  2024cites this paper
• A principled approach to community detection in interareal cortical networks

  2024cites this paper
• Beyond Discrepancy: A Closer Look at the Theory of Distribution Shift

  2024cites this paper
• Domain Generalisation via Imprecise Learning

  2024cites this paper
• On f-Divergence Principled Domain Adaptation: An Improved Framework

  2024influential citation
• Limits of Model Selection under Transfer Learning

  2023cites this paper
• Variational Inference via Rényi Bound Optimization and Multiple-Source Adaptation

  2023cites this paper
• Towards More General Loss and Setting in Unsupervised Domain Adaptation

  2023cites this paper
• When is Importance Weighting Correction Needed for Covariate Shift Adaptation?

  2023cites this paper
• Domain Adaptation via Rebalanced Sub-domain Alignment

  2023cites this paper
• Fairness and Accuracy Under Domain Generalization

  2023cites this paper
• A Variational Formula for Infinity-Rényi Divergence with Applications to Information Leakage

  2022cites this paper
• A new similarity measure for covariate shift with applications to nonparametric regression

  2022cites this paper
• On the Limitations of General Purpose Domain Generalisation Methods

  2022cites this paper
• Improved Stein Variational Gradient Descent with Importance Weights

  2022cites this paper
• A novel domain adaptation theory with Jensen-Shannon divergence

  2022cites this paper
• On-Device Learning for Model Personalization with Large-Scale Cloud-Coordinated Domain Adaption

  2022cites this paper
• Joint Attention-Driven Domain Fusion and Noise-Tolerant Learning for Multi-Source Domain Adaptation

  2022cites this paper
• A belief Rényi divergence for multi-source information fusion and its application in pattern recognition

  2022cites this paper
• Domain-specific feature elimination: multi-source domain adaptation for image classification

  2022cites this paper
• Gated Domain Units for Multi-source Domain Generalization

  2022cites this paper
• A Semantic Consistency Feature Alignment Object Detection Model Based on Mixed-Class Distribution Metrics

  2022cites this paper
• Multi-Domain Adaptation in Neural Machine Translation with Dynamic Sampling Strategies

  2022cites this paper
• Importance Weighting Correction of Regularized Least-Squares for Covariate and Target Shifts

  2022cites this paper
• Convergence of Local Stochastic Gradient Descent

  2022cites this paper
• Revisiting Multi-Domain Machine Translation

  2021cites this paper
• A Discriminative Technique for Multiple-Source Adaptation

  2021influential citation
• A Two-Way alignment approach for unsupervised multi-Source domain adaptation

  2021cites this paper
• Aggregating From Multiple Target-Shifted Sources

  2021cites this paper
• Federated Learning: A signal processing perspective

  2021cites this paper
• Your Classifier can Secretly Suffice Multi-Source Domain Adaptation

  2021influential citation
• Towards a Unified Framework for Learning and Reasoning

  2021influential citation
• Iterative Feature Matching: Toward Provable Domain Generalization with Logarithmic Environments

  2021cites this paper
• Confident Anchor-Induced Multi-Source Free Domain Adaptation

  2021cites this paper
• On Learning Domain-Invariant Representations for Transfer Learning with Multiple Sources

  2021cites this paper
• Universal and data-adaptive algorithms for model selection in linear contextual bandits

  2021cites this paper
• Boosting with Multiple Sources

  2021influential citation
• Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithm

  2021cites this paper
• Estimation and Concentration of Missing Mass of Functions of Discrete Probability Distributions

  2021cites this paper
• Multi-source Few-shot Domain Adaptation

  2021cites this paper
• Missing g-Mass: Investigating the Missing Parts of Distributions

  2021cites this paper
• Multi-source transfer regression via source-target pairwise segment

  2020cites this paper
• Domain Adaptation

  2020cites this paper
• Complexity of Estimating Rényi Entropy of Markov Chains

  2020cites this paper
• On the Value of Target Data in Transfer Learning

  2020cites this paper
• Model Reuse With Reduced Kernel Mean Embedding Specification

  2020cites this paper
• Learn to Expect the Unexpected: Probably Approximately Correct Domain Generalization

  2020cites this paper
• Learning Bounds for Moment-Based Domain Adaptation

  2020cites this paper
• Practical Estimation of Renyi Entropy

  2020cites this paper
• Practical Renyi Entropy Estimation from Birthday Paradox

  2020cites this paper
• Representation Bayesian Risk Decompositions and Multi-Source Domain Adaptation

  2020cites this paper
• Moment-Based Domain Adaptation: Learning Bounds and Algorithms

  2020cites this paper
• Unsupervised Multi-source Domain Adaptation for Regression

  2020cites this paper
• Multiple-source domain adaptation with generative adversarial nets

  2020cites this paper
• A No-Free-Lunch Theorem for MultiTask Learning

  2020cites this paper
• A Theory of Multiple-Source Adaptation with Limited Target Labeled Data

  2020cites this paper
• Beyond H-Divergence: Domain Adaptation Theory With Jensen-Shannon Divergence

  2020cites this paper
• A survey on domain adaptation theory

  2020influential citation
• On generalization in moment-based domain adaptation

  2020cites this paper
• Multiple-Source Adaptation with Domain Classifiers

  2020influential citation
• Explicit Renyi Entropy for Hidden Markov Models

  2020cites this paper
• The Communication-Aware Clustered Federated Learning Problem

  2020influential citation
• A Review of Single-Source Deep Unsupervised Visual Domain Adaptation

  2020cites this paper
• A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization

  2020cites this paper
• Transfer Meta-Learning: Information- Theoretic Bounds and Information Meta-Risk Minimization

  2020cites this paper
• A Computationally Efficient Classification Algorithm in Posterior Drift Model: Phase Transition and Minimax Adaptivity

  2020cites this paper
• Multiple-source adaptation theory and algorithms

  2020influential citation
• Activity recognition in wearables using adversarial multi-source domain adaptation

  2020cites this paper
• FedBoost: Communication-Efficient Algorithms for Federated Learning

  2020influential citation
• On Learning Invariant Representation for Domain Adaptation

  2019cites this paper
• Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Suffices

  2019cites this paper
• Agnostic Federated Learning

  2019cites this paper
• On the analysis of adaptability in multi-source domain adaptation

  2019cites this paper
• Domain Aggregation Networks for Multi-Source Domain Adaptation

  2019influential citation
• Domain Discrepancy Measure for Complex Models in Unsupervised Domain Adaptation

  2019cites this paper
• A review of single-source unsupervised domain adaptation

  2019cites this paper
• Entropies and their Asymptotic Theory in the discrete case

  2019cites this paper
• A Review of Domain Adaptation without Target Labels

  2019cites this paper
• On Learning Invariant Representations for Domain Adaptation

  2019cites this paper
• On Fair Cost Sharing Games in Machine Learning

  2019cites this paper
• Bridging Theory and Algorithm for Domain Adaptation

  2019influential citation
• Revisiting (ε, γ, τ)-similarity learning for domain adaptation

  2018cites this paper
• Generalization Bound for Multiple Source Domain Adaptation

  2018cites this paper
• Multiple Source Domain Adaptation with Adversarial Learning

  2018influential citation
• Unsupervised Domain Adaptation Based on Source-guided Discrepancy

  2018cites this paper
• Marginal Singularity, and the Benefits of Labels in Covariate-Shift

  2018cites this paper
• Algorithms and Theory for Multiple-Source Adaptation

  2018influential citation
• PAC-Bayes and domain adaptation

  2017cites this paper