The Benefit of Multitask Representation Learning

We discuss a general method to learn data representations from multiple tasks. We provide a justification for this method in both settings of multitask learning and learning-to-learn. The method is illustrated in detail in the special case of linear feature learning. Conditions on the theoretical advantage offered by multitask representation learning over independent task learning are established. In particular, focusing on the important example of half-space learning, we derive the regime in which multitask representation learning is beneficial over independent task learning, as a function of the sample size, the number of tasks and the intrinsic data dimensionality. Other potential applications of our results include multitask feature learning in reproducing kernel Hilbert spaces and multilayer, deep networks.

• Publication year

  2015

• Venue

  Journal of machine learning research

• Publication date

  2015-05-23

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1505.06279
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• An Inequality with Applications to Structured Sparsity and Multitask Dictionary Learning

  2014cited by this paper
• Online Multi-Task Learning via Sparse Dictionary Optimization

  2014cited by this paper
• A chain rule for the expected suprema of Gaussian processes

  2014cited by this paper
• Regularization-Based Multitask Learning With Applications to Genome Biology and Biological Imaging

  2014cited by this paper
• A PAC-Bayesian bound for Lifelong Learning

  2013cited by this paper
• Stability and Hypothesis Transfer Learning

  2013cited by this paper
• Multi-task Learning

  2013influential reference
• Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

  2013cited by this paper
• Excess risk bounds for multitask learning with trace norm regularization

  2012influential reference
• Sparse coding for multitask and transfer learning

  2012cited by this paper
• Oracle Inequalities and Optimal Inference under Group Sparsity

  2010cited by this paper
• Linear Algorithms for Online Multitask Classification

  2010cited by this paper
• Regularization Techniques for Learning with Matrices

  2009cited by this paper
• Convex multi-task feature learning

  2008cited by this paper
• A Uniform Lower Error Bound for Half-Space Learning

  2008cited by this paper
• On singular values of matrices with independent rows

  2006cited by this paper
• The Rademacher Complexity of Linear Transformation Classes

  2006cited by this paper
• Bounds for Linear Multi-Task Learning

  2006cited by this paper
• A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data

  2005cited by this paper
• Learning Multiple Tasks with Kernel Methods

  2005cited by this paper
• Rademacher and Gaussian Complexities: Risk Bounds and Structural Results

  2003cited by this paper
• Limitations of Learning Via Embeddings in Euclidean Half Spaces

  2003cited by this paper
• An elementary proof of a theorem of Johnson and Lindenstrauss

  2003cited by this paper
• Exploiting Task Relatedness for Mulitple Task Learning

  2003cited by this paper
• Economic inequality and burden-sharing in the provision of local environmental quality

  2002cited by this paper
• Empirical margin distributions and bounding the generalization error of combined classifiers

  2002cited by this paper
• A Model of Inductive Bias Learning

  2000influential reference
• Learning in Neural Networks: Theoretical Foundations

  1999cited by this paper
• On the sample complexity of PAC learning half-spaces against the uniform distribution

  1995cited by this paper
• Probability Inequalities for Sums of Bounded Random Variables

  1994cited by this paper
• Probability in Banach Spaces: Isoperimetry and Processes

  1991cited by this paper
• A general lower bound on the number of examples needed for learning

  1988cited by this paper
• Probability inequalities for sum of bounded random variables

  1963cited by this paper
• The one-sided barrier problem for Gaussian noise

  1962cited by this paper
• Weak Convergence of Empirical Processes

  year unknowncited by this paper

• Individualised Treatment Effects Estimation with Composite Treatments and Composite Outcomes

  2025cites this paper
• Where and How to Enhance: Discovering Bit-Width Contribution for Mixed Precision Quantization

  2025cites this paper
• What Can We Learn from Inter-Annotator Variability in Skin Lesion Segmentation?

  2025cites this paper
• A Theory for Conditional Generative Modeling on Multiple Data Sources

  2025cites this paper
• Medical multimodal multitask foundation model for lung cancer screening

  2025cites this paper
• HRRPGraphNet++: Dynamic Graph Neural Network with Meta-Learning for Few-Shot HRRP Radar Target Recognition

  2025cites this paper
• Transformers are almost optimal metalearners for linear classification

  2025cites this paper
• Prompt and Parameter Co-Optimization for Large Language Models

  2025cites this paper
• Multitask Causal Contrastive Learning

  2025cites this paper
• Representation Transfer Learning via Multiple Pre-Trained Models for Linear Regression

  2025cites this paper
• Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning

  2025cites this paper
• Towards Understanding the Benefit of Multitask Representation Learning in Decision Process

  2025cites this paper
• Reasoning without Regret

  2025cites this paper
• Distribution-dependent Generalization Bounds for Tuning Linear Regression Across Tasks

  2025influential citation
• Open-Set Domain Adaptation Under Background Distribution Shift: Challenges and A Provably Efficient Solution

  2025cites this paper
• Cross-Learning from Scarce Data via Multi-Task Constrained Optimization

  2025cites this paper
• Multi-task meta-initialized DQN for fast adaptation to unseen slicing tasks in O-RAN

  2025cites this paper
• Robust and Adaptive Spectral Method for Representation Multi-Task Learning with Contamination

  2025cites this paper
• Efficient Morphology-Aware Policy Transfer to New Embodiments

  2025cites this paper
• GeoERM: Geometry-Aware Multi-Task Representation Learning on Riemannian Manifolds

  2025cites this paper
• Drift-aware Collaborative Assistance Mixture of Experts for Heterogeneous Multistream Learning

  2025influential citation
• Fast and robust invariant generalized linear models.

  2025cites this paper
• Meta-learning of shared linear representations beyond well-specified linear regression

  2025influential citation
• Statistical Deficiency for Task Inclusion Estimation

  2025cites this paper
• Provable Sample-Efficient Transfer Learning Conditional Diffusion Models via Representation Learning

  2025cites this paper
• Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions

  2025cites this paper
• Model Merging via Multi-Teacher Knowledge Distillation

  2025cites this paper
• Transfer Learning through Enhanced Sufficient Representation: Enriching Source Domain Knowledge with Target Data

  2025cites this paper
• Safe Screening Rules for Group OWL Models

  2025cites this paper
• PAL: Sample-Efficient Personalized Reward Modeling for Pluralistic Alignment

  2025cites this paper
• Clustering-Based Weight Orthogonalization for Stabilizing Deep Reinforcement Learning

  2025cites this paper
• Representative Action Selection for Large Action Space: From Bandits to MDPs

  2025cites this paper
• How Many Domains Suffice for Domain Generalization? A Tight Characterization via the Domain Shattering Dimension

  2025cites this paper
• DEMONSTRATE: Zero-shot Language to Robotic Control via Multi-task Demonstration Learning

  2025cites this paper
• MM-RunAssist: mmWave-based Respiratory and Running Rhythm Analysis during Treadmill Workouts

  2025cites this paper
• Bridging Lifelong and Multi-Task Representation Learning via Algorithm and Complexity Measure

  2025cites this paper
• Integrative tensor regression for stratified data with application to neuroimaging analysis

  2025cites this paper
• α-LoRA: Effective Fine-Tuning via Base Model Rescaling

  2025cites this paper
• DivMerge: A divergence-based model merging method for multi-tasking

  2025cites this paper
• Provable Multi-Party Reinforcement Learning with Diverse Human Feedback

  2024cites this paper
• UNITE: Multitask Learning With Sufficient Feature for Dense Prediction

  2024cites this paper
• Parental status and markers of brain and cellular age: A 3D convolutional network and classification study.

  2024cites this paper
• A New Neural Kernel Regime: The Inductive Bias of Multi-Task Learning

  2024cites this paper
• Collaborative Learning with Shared Linear Representations: Statistical Rates and Optimal Algorithms

  2024cites this paper
• On the Stability and Generalization of Meta-Learning

  2024cites this paper
• Beyond Discrepancy: A Closer Look at the Theory of Distribution Shift

  2024cites this paper
• Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks

  2024cites this paper
• Can Large Language Models Generate High-quality Patent Claims?

  2024cites this paper
• Rethinking Meta-Learning from a Learning Lens

  2024cites this paper
• Learning to Learn with Contrastive Meta-Objective

  2024cites this paper
• The Effects of Multi-Task Learning on ReLU Neural Network Functions

  2024cites this paper
• Patent-CR: A Dataset for Patent Claim Revision

  2024cites this paper
• Hierarchical representation learning for next basket recommendation

  2024cites this paper
• Enhancing Accuracy in Generative Models via Knowledge Transfer

  2024cites this paper
• Scaling laws for learning with real and surrogate data

  2024cites this paper
• Memorization Capacity for Additive Fine-Tuning with Small ReLU Networks

  2024cites this paper
• Deep Pareto Reinforcement Learning for Multi-Objective Recommender Systems

  2024cites this paper
• A Location-Independent Human Activity Recognition Method Based on CSI: System, Architecture, Implementation

  2024cites this paper
• Entity Resolution in Situated Dialog With Unimodal and Multimodal Transformers

  2024cites this paper
• Disentangling and Mitigating the Impact of Task Similarity for Continual Learning

  2024cites this paper
• A CMDP-within-online framework for Meta-Safe Reinforcement Learning

  2024cites this paper
• Natural language processing in the patent domain: a survey

  2024cites this paper
• The Power of Active Multi-Task Learning in Reinforcement Learning from Human Feedback

  2024cites this paper
• Fast and Sample Efficient Multi-Task Representation Learning in Stochastic Contextual Bandits

  2024cites this paper
• Task Diversity Shortens the In-Context Learning Plateau

  2024cites this paper
• Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks

  2024cites this paper
• Manifold Learning via Foliations and Knowledge Transfer

  2024cites this paper
• On the Value of Head Labels in Multi-Label Text Classification

  2024cites this paper
• Few-shot Multi-Task Learning of Linear Invariant Features with Meta Subspace Pursuit

  2024cites this paper
• Semi-supervised Group DRO: Combating Sparsity with Unlabeled Data

  2024cites this paper
• Learning with Shared Representations: Statistical Rates and Efficient Algorithms

  2024cites this paper
• Efficient and generalizable prediction of molecular alterations in multiple cancer cohorts using H&E whole slide images

  2024cites this paper
• A survey on kernel-based multi-task learning

  2024cites this paper
• Task Diversity Shortens the ICL Plateau

  2024cites this paper
• An Information-Theoretic Analysis of In-Context Learning

  2024cites this paper
• Adaptive Fine-Tuning in Degradation-Time-Series Forecasting via Generating Source Domain

  2024cites this paper
• One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion

  2024influential citation
• Any-Way Meta Learning

  2024cites this paper
• Dynamic Graph Embedding via Meta-Learning

  2024cites this paper
• More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithms

  2024cites this paper
• Guarantees for Nonlinear Representation Learning: Non-identical Covariates, Dependent Data, Fewer Samples

  2024cites this paper
• Exploiting Feature Heterogeneity for Improved Generalization in Federated Multi-task Learning

  2023cites this paper
• Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspective

  2023cites this paper
• A Theory of Multimodal Learning

  2023cites this paper
• Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Data

  2023cites this paper
• Multitask Learning with No Regret: from Improved Confidence Bounds to Active Learning

  2023cites this paper
• Estimation of Models with Limited Data by Leveraging Shared Structure

  2023cites this paper
• Nonlinear Meta-Learning Can Guarantee Faster Rates

  2023influential citation
• Kernel-based learning of orthogonal functions

  2023cites this paper
• Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks

  2023cites this paper
• Multilinear multitask learning by transformed tensor singular value decomposition

  2023cites this paper
• Implicit regularization of multi-task learning and finetuning in overparameterized neural networks

  2023cites this paper
• Representation Transfer Learning via Multiple Pre-trained models for Linear Regression

  2023cites this paper
• Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse

  2023cites this paper
• DAC-MR: Data Augmentation Consistency Based Meta-Regularization for Meta-Learning

  2023cites this paper
• Neural Processes with Stability

  2023cites this paper
• Mastering Long-Tail Complexity on Graphs: Characterization, Learning, and Generalization

  2023cites this paper
• Dynamic Inter-treatment Information Sharing for Individualized Treatment Effects Estimation

  2023cites this paper
• Shared dynamics learning for large-scale traveling salesman problem

  2023cites this paper
• Best-effort adaptation

  2023cites this paper