Desiderata for Representation Learning: A Causal Perspective

Representation learning constructs low-dimensional representations to summarize essential features of high-dimensional data. This learning problem is often approached by describing various desiderata associated with learned representations; e.g., that they be non-spurious, efficient, or disentangled. It can be challenging, however, to turn these intuitive desiderata into formal criteria that can be measured and enhanced based on observed data. In this paper, we take a causal perspective on representation learning, formalizing non-spuriousness and efficiency (in supervised representation learning) and disentanglement (in unsupervised representation learning) using counterfactual quantities and observable consequences of causal assertions. This yields computable metrics that can be used to assess the degree to which representations satisfy the desiderata of interest and learn non-spurious and disentangled representations from single observational datasets.

• Publication year

  2021

• Venue

  arXiv.org

• Publication date

  2021-09-08

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 2109.03795
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Causal Inference with Latent Treatments

  2021cited by this paper
• Intact-VAE: Estimating Treatment Effects under Unobserved Confounding

  2021cited by this paper
• A Proxy Variable View of Shared Confounding

  2021cited by this paper
• Predictive Modeling in the Presence of Nuisance-Induced Spurious Correlations

  2021cited by this paper
• Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests

  2021cited by this paper
• Causes of Effects: Learning individual responses from population data

  2021cited by this paper
• Local Explanations via Necessity and Sufficiency: Unifying Theory and Practice

  2021cited by this paper
• Explaining Black-Box Algorithms Using Probabilistic Contrastive Counterfactuals

  2021cited by this paper
• Identifying spurious correlations for robust text classification

  2020influential reference
• Disentangled Generative Causal Representation Learning

  2020cited by this paper
• CausalVAE: Disentangled Representation Learning via Neural Structural Causal Models

  2020cited by this paper
• Is Independence all you need? On the Generalization of Representations Learned from Correlated Data

  2020cited by this paper
• A Calculus for Stochastic Interventions: Causal Effect Identification and Surrogate Experiments

  2020influential reference
• Towards Clarifying the Theory of the Deconfounder

  2020influential reference
• A Simple Framework for Contrastive Learning of Visual Representations

  2020influential reference
• Generalization Bounds and Representation Learning for Estimation of Potential Outcomes and Causal Effects

  2020cited by this paper
• Weakly-Supervised Disentanglement Without Compromises

  2020cited by this paper
• Environment Inference for Invariant Learning

  2020cited by this paper
• Robustness to Spurious Correlations in Text Classification via Automatically Generated Counterfactuals

  2020influential reference
• Causal Estimation with Functional Confounders

  2020influential reference
• Invariant Representation Learning for Treatment Effect Estimation

  2020cited by this paper
• Towards Unifying Feature Attribution and Counterfactual Explanations: Different Means to the Same End

  2020cited by this paper
• Underspecification Presents Challenges for Credibility in Modern Machine Learning

  2020cited by this paper
• Representation Learning via Invariant Causal Mechanisms

  2020cited by this paper
• Using Embeddings to Correct for Unobserved Confounding

  2019cited by this paper
• The seven tools of causal inference, with reflections on machine learning

  2019influential reference
• Support and Invertibility in Domain-Invariant Representations

  2019cited by this paper
• On Multi-Cause Approaches to Causal Inference with Unobserved Counfounding: Two Cautionary Failure Cases and A Promising Alternative

  2019cited by this paper
• Disentangling Factors of Variation Using Few Labels

  2019cited by this paper
• On Learning Invariant Representations for Domain Adaptation

  2019cited by this paper
• Invariant Risk Minimization

  2019influential reference
• Variational Autoencoders and Nonlinear ICA: A Unifying Framework

  2019cited by this paper
• Group-based Learning of Disentangled Representations with Generalizability for Novel Contents

  2019cited by this paper
• Weakly Supervised Disentanglement with Guarantees

  2019cited by this paper
• Deep causal representation learning for unsupervised domain adaptation

  2019cited by this paper
• Disentangled Representation Learning with Wasserstein Total Correlation

  2019cited by this paper
• Comment: The Challenges of Multiple Causes

  2019cited by this paper
• Adapting Text Embeddings for Causal Inference

  2019cited by this paper
• Isolating Sources of Disentanglement in VAEs

  2018cited by this paper
• Invariant Representations without Adversarial Training

  2018cited by this paper
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations

  2018cited by this paper
• Generalization in anti-causal learning

  2018cited by this paper
• Disentangling by Factorising

  2018influential reference
• Disentangling the independently controllable factors of variation by interacting with the world

  2018cited by this paper
• Multiple Causal Inference with Latent Confounding

  2018cited by this paper
• Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness

  2018influential reference
• The Blessings of Multiple Causes

  2018influential reference
• Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations

  2017cited by this paper
• Semi-Parametric Causal Sufficient Dimension Reduction Of High Dimensional Treatments

  2017cited by this paper
• Causal Structure Learning

  2017cited by this paper
• Overlap in observational studies with high-dimensional covariates

  2017cited by this paper
• Graph-based Isometry Invariant Representation Learning

  2017cited by this paper
• Variational Inference of Disentangled Latent Concepts from Unlabeled Observations

  2017cited by this paper
• Emergence of Invariance and Disentanglement in Deep Representations

  2017cited by this paper
• Learning Independent Causal Mechanisms

  2017cited by this paper
• Frequentist Consistency of Variational Bayes

  2017influential reference
• Feature Selection as Causal Inference: Experiments with Text Classification

  2017cited by this paper
• Structured Latent Factor Analysis for Large-scale Data: Identifiability, Estimability, and Their Implications

  2017influential reference
• Why Are Big Data Matrices Approximately Low Rank?

  2017cited by this paper
• Causal Invariance as an Essential Constraint for Creating a Causal Representation of the World

  2017cited by this paper
• Causal feature learning: an overview

  2017cited by this paper
• Independently Controllable Factors

  2017cited by this paper
• beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

  2016cited by this paper
• Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

  2016cited by this paper
• Variational Inference: A Review for Statisticians

  2016cited by this paper
• Maximum Likelihood Estimation and Inference for Approximate Factor Models of High Dimension

  2016cited by this paper
• Discovery of Treatments from Text Corpora

  2016cited by this paper
• Semi-supervised interpolation in an anticausal learning scenario

  2015cited by this paper
• Causal and anti-causal learning in pattern recognition for neuroimaging

  2015cited by this paper
• Visual Causal Feature Learning

  2014cited by this paper
• Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models

  2014cited by this paper
• Deep Learning Face Attributes in the Wild

  2014influential reference
• Semi-supervised Learning in Causal and Anticausal Settings

  2013cited by this paper
• Auto-Encoding Variational Bayes

  2013influential reference
• Why ask why? Forward causal inference and reverse causal questions

  2013cited by this paper
• The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]

  2012influential reference
• Representation Learning: A Review and New Perspectives

  2012cited by this paper
• On causal and anticausal learning

  2012cited by this paper
• Latent aspect rating analysis without aspect keyword supervision

  2011cited by this paper
• Latent aspect rating analysis on review text data: a rating regression approach

  2010cited by this paper
• Latent Dirichlet Allocation

  2009cited by this paper
• Interventions and Causal Inference

  2007influential reference
• Mixed Membership Stochastic Blockmodels

  2007cited by this paper
• Dimensionality Reduction by Learning an Invariant Mapping

  2006cited by this paper
• Bayesian Mixed Membership Models for Soft Clustering and Classification

  2004cited by this paper
• Inference of population structure using multilocus genotype data.

  2000cited by this paper
• Probabilities of causation: Bounds and identification

  2000cited by this paper
• Probabilistic Principal Component Analysis

  1999cited by this paper
• POSTERIOR PREDICTIVE ASSESSMENT OF MODEL FITNESS VIA REALIZED DISCREPANCIES

  1996cited by this paper
• Markov chain Monte Carlo in Practice

  1996cited by this paper
• Causal diagrams for empirical research

  1995cited by this paper
• Mixture models : inference and applications to clustering

  1989cited by this paper
• Rank degeneracy and least squares problems

  1976cited by this paper
• Causality : Models , Reasoning , and Inference

  year unknowninfluential reference

• A Novel Multi-task Causal Representation Learning Approach for Interpretable Maritime Collision Severity Prediction

  2026cites this paper
• Decoupling solvent features to address spurious correlations in ceramic Organic Solvent Nanofiltration membranes

  2026cites this paper
• Seeking Necessary and Sufficient Information from Multimodal Medical Data

  2026cites this paper
• Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders

  2025cites this paper
• Towards Interpretable Deep Generative Models via Causal Representation Learning

  2025cites this paper
• Causal perception inspired representation learning for trustworthy image quality assessment

  2025cites this paper
• Compositional Causal Reasoning Evaluation in Language Models

  2025cites this paper
• Policy Learning with a Natural Language Action Space: A Causal Approach

  2025cites this paper
• Debiasing Reward Models by Representation Learning with Guarantees

  2025cites this paper
• Learning Robust Intervention Representations with Delta Embeddings

  2025cites this paper
• Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning

  2025cites this paper
• The Intrinsic Dimension of Collider Events and Model-Independent Searches in 100 Dimensions

  2025cites this paper
• Sparse, self-organizing ensembles of local kernels detect rare statistical anomalies

  2025cites this paper
• Medical Image Quality Assessment Based on Probability of Necessity and Sufficiency

  2024cites this paper
• Pioneering new paths: the role of generative modelling in neurological disease research

  2024cites this paper
• Causal Representation Learning with Generative Artificial Intelligence: Application to Texts as Treatments

  2024influential citation
• Genetic Architectures of Medical Images Revealed by Registration of Multiple Modalities

  2024cites this paper
• Deep Autoregressive Models as Causal Inference Engines

  2024cites this paper
• CSRec: Rethinking Sequential Recommendation from A Causal Perspective.

  2024cites this paper
• A Unified Causal View of Instruction Tuning

  2024cites this paper
• The Essential Role of Causality in Foundation World Models for Embodied AI

  2024cites this paper
• On the Challenges and Opportunities in Generative AI

  2024cites this paper
• Revisiting Disentanglement in Downstream Tasks: A Study on Its Necessity for Abstract Visual Reasoning

  2024cites this paper
• Seeking the Sufficiency and Necessity Causal Features in Multimodal Representation Learning

  2024cites this paper
• Towards the Causal Complete Cause of Multi-Modal Representation Learning

  2024cites this paper
• Disentangled Representations for Causal Cognition

  2024influential citation
• Causal Perception Inspired Representation Learning for Trustworthy Image Quality Assessment

  2024cites this paper
• GIFT: A Framework Towards Global Interpretable Faithful Textual Explanations of Vision Classifiers

  2024cites this paper
• Causal Representation Learning for GAN-Generated Face Image Quality Assessment

  2024cites this paper
• Transportable Representations for Domain Generalization

  2024cites this paper
• Causal Inference for Human-Language Model Collaboration

  2024cites this paper
• Identifiable Latent Bandits: Leveraging observational data for personalized decision-making

  2024cites this paper
• Recursive Causal Discovery

  2024cites this paper
• Identifying General Mechanism Shifts in Linear Causal Representations

  2024cites this paper
• Active and Passive Causal Inference Learning

  2023cites this paper
• On Learning Necessary and Sufficient Causal Graphs

  2023influential citation
• Causality-Aware Channel State Information Encoding

  2023cites this paper
• CAT: Causal Audio Transformer for Audio Classification

  2023influential citation
• Counterfactual Learning on Graphs: A Survey

  2023cites this paper
• A Measure-Theoretic Axiomatisation of Causality

  2023cites this paper
• Causal Component Analysis

  2023cites this paper
• Nonparametric Identifiability of Causal Representations from Unknown Interventions

  2023cites this paper
• Towards Trustworthy Explanation: On Causal Rationalization

  2023influential citation
• On the Identifiability of Quantized Factors

  2023cites this paper
• Causal Reinforcement Learning: A Survey

  2023cites this paper
• Time series prediction and causation

  2023cites this paper
• Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation

  2023cites this paper
• Genetic Architectures of Medical Images Revealed by Registration and Fusion of Multiple Modalities

  2023cites this paper
• Specify Robust Causal Representation from Mixed Observations

  2023influential citation
• Causal SAR ATR with Limited Data via Dual Invariance

  2023cites this paper
• Invariant Learning via Probability of Sufficient and Necessary Causes

  2023cites this paper
• Learning Invariant Representations with a Nonparametric Nadaraya-Watson Head

  2023cites this paper
• A method to assess trustworthiness of machine coding at scale

  2023cites this paper
• From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling

  2023cites this paper
• C-Disentanglement: Discovering Causally-Independent Generative Factors under an Inductive Bias of Confounder

  2023influential citation
• Causal Context Connects Counterfactual Fairness to Robust Prediction and Group Fairness

  2023cites this paper
• Self-Supervised Disentanglement by Leveraging Structure in Data Augmentations

  2023cites this paper
• Identifiability of Discretized Latent Coordinate Systems via Density Landmarks Detection

  2023cites this paper
• Partial Disentanglement with Partially-Federated GANs (PaDPaF)

  2022cites this paper
• Indeterminacy in Generative Models: Characterization and Strong Identifiability

  2022cites this paper
• Towards Cross-Modal Causal Structure and Representation Learning

  2022cites this paper
• Improving Generalization via Uncertainty Driven Perturbations

  2022cites this paper
• On Pitfalls of Identifiability in Unsupervised Learning. A Note on: "Desiderata for Representation Learning: A Causal Perspective"

  2022influential citation
• Generative multitask learning mitigates target-causing confounding

  2022cites this paper
• INDETERMINACY AND STRONG IDENTIFIABILITY IN GENERATIVE MODELS

  2022cites this paper
• Towards efficient representation identification in supervised learning

  2022cites this paper
• Learning Latent Structural Causal Models

  2022cites this paper
• Are All Spurious Features in Natural Language Alike? An Analysis through a Causal Lens

  2022influential citation
• PaDPaF: Partial Disentanglement with Partially-Federated GANs

  2022cites this paper
• On Causal Rationalization

  2022influential citation
• CPL: Counterfactual Prompt Learning for Vision and Language Models

  2022cites this paper
• Disentanglement of Correlated Factors via Hausdorff Factorized Support

  2022influential citation
• Causal Class Activation Maps for Weakly-Supervised Semantic Segmentation

  2022cites this paper
• Learning Causal Representations with Granger PCA

  2022cites this paper
• Interventional Causal Representation Learning

  2022influential citation
• Bias Challenges in Counterfactual Data Augmentation

  2022cites this paper
• Towards Benchmarking Explainable Artificial Intelligence Methods

  2022cites this paper
• A SYMMETRY L EARNING FOR C OUNTERFACTUAL - I NVARIANT C LASSIFICATION IN OOD T ASKS

  2022cites this paper
• Sampling Through the Lens of Sequential Decision Making

  2022cites this paper
• Data-Centric Epidemic Forecasting: A Survey

  2022cites this paper
• COEM: Cross-Modal Embedding for MetaCell Identification

  2022cites this paper
• A structural characterization of shortcut features for prediction

  2022cites this paper
• Invariant and Transportable Representations for Anti-Causal Domain Shifts

  2022cites this paper
• Towards a Grounded Theory of Causation for Embodied AI

  2022cites this paper
• Indeterminacy in Latent Variable Models: Characterization and Strong Identifiability

  2022cites this paper
• DagSim: Combining DAG-based model structure with unconstrained data types and relations for flexible, transparent, and modularized data simulation

  2022cites this paper
• Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation

  2022cites this paper
• Generalizable Information Theoretic Causal Representation

  2022cites this paper
• Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspective

  2021cites this paper
• Transportable Representations for Out-of-distribution Generalization

  year unknowncites this paper
• Physics of Life Reviews

  year unknowninfluential citation