Personalized Federated Learning through Local Memorization

Federated learning allows clients to collaboratively learn statistical models while keeping their data local. Federated learning was originally used to train a unique global model to be served to all clients, but this approach might be sub-optimal when clients' local data distributions are heterogeneous. In order to tackle this limitation, recent personalized federated learning methods train a separate model for each client while still leveraging the knowledge available at other clients. In this work, we exploit the ability of deep neural networks to extract high quality vectorial representations (embeddings) from non-tabular data, e.g., images and text, to propose a personalization mechanism based on local memorization. Personalization is obtained by interpolating a collectively trained global model with a local $k$-nearest neighbors (kNN) model based on the shared representation provided by the global model. We provide generalization bounds for the proposed approach in the case of binary classification, and we show on a suite of federated datasets that this approach achieves significantly higher accuracy and fairness than state-of-the-art methods.

• Publication year

  2021

• Venue

  International Conference on Machine Learning

• Publication date

  2021-11-17

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 2111.09360
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• On Bridging Generic and Personalized Federated Learning for Image Classification

  2022cited by this paper
• FedProto: Federated Prototype Learning across Heterogeneous Clients

  2021cited by this paper
• FedZKT: Zero-Shot Knowledge Transfer towards Heterogeneous On-Device Models in Federated Learning

  2021cited by this paper
• Fine-tuning is Fine in Federated Learning

  2021cited by this paper
• Federated Multi-Task Learning under a Mixture of Distributions

  2021cited by this paper
• Debiasing Model Updates for Improving Personalized Federated Training

  2021cited by this paper
• Personalized Federated Learning with Gaussian Processes

  2021cited by this paper
• Data-Free Knowledge Distillation for Heterogeneous Federated Learning

  2021cited by this paper
• FedProto: Federated Prototype Learning over Heterogeneous Devices

  2021cited by this paper
• Personalized Federated Learning using Hypernetworks

  2021influential reference
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout

  2021cited by this paper
• Exploiting Shared Representations for Personalized Federated Learning

  2021influential reference
• FedU: A Unified Framework for Federated Multi-Task Learning with Laplacian Regularization

  2021cited by this paper
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients

  2020cited by this paper
• Personalized Federated Learning with First Order Model Optimization

  2020cited by this paper
• Personalized Cross-Silo Federated Learning on Non-IID Data

  2020cited by this paper
• Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach

  2020cited by this paper
• Ditto: Fair and Robust Federated Learning Through Personalization

  2020influential reference
• Think Locally, Act Globally: Federated Learning with Local and Global Representations

  2020cited by this paper
• Lower Bounds and Optimal Algorithms for Personalized Federated Learning

  2020cited by this paper
• Simple, Efficient and Convenient Decentralized Multi-task Learning for Neural Networks

  2020cited by this paper
• Salvaging Federated Learning by Local Adaptation

  2020cited by this paper
• Federated Learning of a Mixture of Global and Local Models

  2020influential reference
• Federated Learning with Matched Averaging

  2020cited by this paper
• Three Approaches for Personalization with Applications to Federated Learning

  2020influential reference
• Adaptive Federated Optimization

  2020cited by this paper
• Adaptive Personalized Federated Learning

  2020cited by this paper
• An Efficient Framework for Clustered Federated Learning

  2020cited by this paper
• Ensemble Distillation for Robust Model Fusion in Federated Learning

  2020cited by this paper
• Personalized Federated Learning with Moreau Envelopes

  2020cited by this paper
• Nearest Neighbor Machine Translation

  2020cited by this paper
• Advances and Open Problems in Federated Learning

  2019cited by this paper
• Agnostic Federated Learning

  2019cited by this paper
• Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs

  2019cited by this paper
• Adaptive Gradient-Based Meta-Learning Methods

  2019cited by this paper
• Variational Federated Multi-Task Learning

  2019cited by this paper
• Federated Learning: Challenges, Methods, and Future Directions

  2019cited by this paper
• An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models

  2019cited by this paper
• Improving Federated Learning Personalization via Model Agnostic Meta Learning

  2019cited by this paper
• Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints

  2019cited by this paper
• FedMD: Heterogenous Federated Learning via Model Distillation

  2019cited by this paper
• Federated Learning With Differential Privacy: Algorithms and Performance Analysis

  2019cited by this paper
• SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning

  2019influential reference
• Generalization through Memorization: Nearest Neighbor Language Models

  2019influential reference
• Federated Learning with Personalization Layers

  2019influential reference
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning

  2019cited by this paper
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning

  2018cited by this paper
• MobileNetV2: Inverted Residuals and Linear Bottlenecks

  2018cited by this paper
• On the Convergence of Federated Optimization in Heterogeneous Networks

  2018cited by this paper
• LEAF: A Benchmark for Federated Settings

  2018cited by this paper
• A Simple Cache Model for Image Recognition

  2018influential reference
• ProtoNN: Compressed and Accurate kNN for Resource-scarce Devices

  2017cited by this paper
• Billion-Scale Similarity Search with GPUs

  2017cited by this paper
• Prototypical Networks for Few-shot Learning

  2017influential reference
• Federated Multi-Task Learning

  2017cited by this paper
• Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs

  2016cited by this paper
• Federated Learning: Strategies for Improving Communication Efficiency

  2016cited by this paper
• Communication-Efficient Learning of Deep Networks from Decentralized Data

  2016influential reference
• Decentralized Collaborative Learning of Personalized Models over Networks

  2016cited by this paper
• Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets

  2015cited by this paper
• Understanding Machine Learning From Theory to Algorithms 1st Edition Shwartz Solutions Manual

  2015cited by this paper
• Learning to Transduce with Unbounded Memory

  2015cited by this paper
• Learning Multiple Layers of Features from Tiny Images

  2009influential reference
• Pachinko allocation: DAG-structured mixture models of topic correlations

  2006cited by this paper
• Efficient and Adaptive Estimation for Semiparametric Models.

  1994cited by this paper

• Federated Ensemble Learning with Progressive Model Personalization

  2026cites this paper
• FedHypeVAE: Federated Learning with Hypernetwork Generated Conditional VAEs for Differentially Private Embedding Sharing

  2026cites this paper
• Achieving Machine Learning Dependability Through Model Switching and Compression

  2026cites this paper
• Per-FedDMA: A personalized federated learning method based on deep multisampling and hypernetwork dynamic adaptation

  2026cites this paper
• FedDFS: Data filtering for stable federated learning

  2025cites this paper
• FedBM: Balancing models for personalized federated learning

  2025cites this paper
• DynFed: Adaptive Federated Learning via Quantization-Aware Knowledge Distillation

  2025cites this paper
• KApFL: Kolmogorov-Arnold Networks for Personalized Federated Learning

  2025cites this paper
• SVQ-VAE: Federated-Learning-Based Semantic-Aware Communication for Vehicular Networks

  2025cites this paper
• FSPFL: Mitigating Communication Gap in Personalization Federated Learning Through Flexible Sparsity Allocation

  2025cites this paper
• DSPFL: A Deep-Layer Sign Sharing Personalized Federated Learning Scheme for Mitigating Poisoning Attacks

  2025cites this paper
• Generalizable Federated Learning using Client Adaptive Focal Modulation

  2025cites this paper
• Knowledge-distillation based personalized federated learning with distribution constraints

  2025cites this paper
• FeDaL: Federated Dataset Learning for Time Series Foundation Models

  2025cites this paper
• FedFOC: Personalized Federated Learning via Fine-Grained One-Shot Clustering

  2025cites this paper
• Embedding-Based Federated Data Sharing via Differentially Private Conditional VAEs

  2025cites this paper
• FedOne: Query-Efficient Federated Learning for Black-box Discrete Prompt Learning

  2025cites this paper
• Federated Learning with Unlabeled Clients: Personalization Can Happen in Low Dimensions

  2025cites this paper
• Adaptive Huber Based Federated Learning for Sparse Optimization

  2025cites this paper
• Event-Driven Online Vertical Federated Learning

  2025cites this paper
• Hot-pluggable Federated Learning: Bridging General and Personalized FL via Dynamic Selection

  2025cites this paper
• Recent Advances in Federated Learning for Connected Autonomous Vehicles: Addressing Privacy, Performance, and Scalability Challenges

  2025cites this paper
• Federated Unlearning Made Practical: Seamless Integration via Negated Pseudo-Gradients

  2025cites this paper
• pFedCal: Lightweight Personalized Federated Learning With Adaptive Calibration Strategy

  2025cites this paper
• Towards Personalized Federated Learning via Contrastive-Augmented Local Memorization Retrieval

  2025cites this paper
• dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Analysis

  2025cites this paper
• Personalized federated learning via decoupling self-knowledge distillation and global adaptive aggregation

  2025cites this paper
• Individualized Data Generation in Personalized Federated Learning

  2025cites this paper
• FedPnP: Personalized graph-structured federated learning

  2025cites this paper
• GHPFL: Advancing Personalized Edge-Based Learning Through Optimized Bandwidth Utilization

  2025cites this paper
• Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead

  2025cites this paper
• Federated Learning for 6G Networks: Navigating Privacy Benefits and Challenges

  2025cites this paper
• FedPDN: Personalized Federated Learning With Interclass Similarity Constraint for Medical Image Classification Through Parameter Decoupling

  2025cites this paper
• Applied Federated Model Personalization in the Industrial Domain: A Comparative Study

  2025cites this paper
• Bayesian Coreset Optimization for Personalized Federated Learning

  2025cites this paper
• Heterogeneity-aware Task Scheduling based on Personalized Federated Reinforcement Learning

  2025cites this paper
• Incentivizing Truthful Collaboration in Heterogeneous Federated Learning

  2024cites this paper
• FedMeS: Personalized Federated Continual Learning Leveraging Local Memory

  2024cites this paper
• Tuning DNN Model Compression to Resource and Data Availability in Cooperative Training

  2024cites this paper
• Gradient Coreset for Federated Learning

  2024cites this paper
• MetaVers: Meta-Learned Versatile Representations for Personalized Federated Learning

  2024influential citation
• Learn What You Need in Personalized Federated Learning

  2024cites this paper
• Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks

  2024cites this paper
• Dependable Distributed Training of Compressed Machine Learning Models

  2024cites this paper
• Global and Local Prompts Cooperation via Optimal Transport for Federated Learning

  2024cites this paper
• A comprehensive survey of federated transfer learning: challenges, methods and applications

  2024cites this paper
• TransFed: A way to epitomize Focal Modulation using Transformer-based Federated Learning

  2024cites this paper
• MiniPFL: Mini federations for hierarchical personalized federated learning

  2024cites this paper
• Late to the party? On-demand unlabeled personalized federated learning

  2024cites this paper
• FedGCR: Achieving Performance and Fairness for Federated Learning with Distinct Client Types via Group Customization and Reweighting

  2024cites this paper
• Personalized Federated Learning with Spurious Features: An Adversarial Approach

  2024cites this paper
• Context-Aware Orchestration of Energy-Efficient Gossip Learning Schemes

  2024cites this paper
• MH-pFLID: Model Heterogeneous personalized Federated Learning via Injection and Distillation for Medical Data Analysis

  2024cites this paper
• Balancing Similarity and Complementarity for Federated Learning

  2024cites this paper
• FedCache 2.0: Exploiting the Potential of Distilled Data in Knowledge Cache-driven Federated Learning

  2024influential citation
• Decentralized Personalized Federated Learning

  2024influential citation
• Communication-Efficient and Privacy-Preserving Decentralized Meta-Learning

  2024cites this paper
• Bridging Model Heterogeneity in Federated Learning via Uncertainty-based Asymmetrical Reciprocity Learning

  2024cites this paper
• Enhancing the Generalization of Personalized Federated Learning with Multi-head Model and Ensemble Voting

  2024influential citation
• Federated brain tumor segmentation: An extensive benchmark

  2024cites this paper
• Feed: Towards Personalization-Effective Federated Learning

  2024cites this paper
• Fine-grained personalized federated learning via transformer in the transformer framework

  2024cites this paper
• Adaptive Personalized Federated Learning for Non-IID Data with Continual Distribution Shift

  2024cites this paper
• Exploiting Complex Network-Based Clustering for Personalization-Enhanced Hierarchical Federated Edge Learning

  2024cites this paper
• FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees

  2024cites this paper
• Applied Federated Model Personalisation in the Industrial Domain: A Comparative Study

  2024cites this paper
• Customized Federated Kernel Regression learning for predicting natural gas hydrate equilibrium with thermodynamic inhibitors: A comprehensive study

  2024cites this paper
• Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration

  2024cites this paper
• FedCache 2.0: Federated Edge Learning with Knowledge Caching and Dataset Distillation

  2024influential citation
• FairDPFL-SCS: Fair Dynamic Personalized Federated Learning with strategic client selection for improved accuracy and fairness

  2024cites this paper
• FedAli: Personalized Federated Learning Alignment with Prototype Layers for Generalized Mobile Services

  2024influential citation
• Asymmetric Mutual Learning for Decentralized Federated Medical Imaging

  2024cites this paper
• Personalized client-edge-cloud hierarchical federated learning in mobile edge computing

  2024cites this paper
• Personalized federated few-shot node classification

  2024cites this paper
• Personalized Federated Learning with Collaborative Aggregation Networks for Multi-Site Brain Disorder Diagnosis

  2024cites this paper
• Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuning

  2023cites this paper
• FedSoL: Bridging Global Alignment and Local Generality in Federated Learning

  2023cites this paper
• Provably Personalized and Robust Federated Learning

  2023cites this paper
• PeFLL: Personalized Federated Learning by Learning to Learn

  2023cites this paper
• Principled Federated Domain Adaptation: Gradient Projection and Auto-Weighting

  2023influential citation
• Federated Variational Inference: Towards Improved Personalization and Generalization

  2023influential citation
• Confidence-Aware Personalized Federated Learning via Variational Expectation Maximization

  2023cites this paper
• A Survey of Federated Evaluation in Federated Learning

  2023cites this paper
• FedDWA: Personalized Federated Learning with Dynamic Weight Adjustment

  2023cites this paper
• Federated Learning of Shareable Bases for Personalization-Friendly Image Classification

  2023cites this paper
• Personalized Fair Split Learning for Resource-Constrained Internet of Things

  2023cites this paper
• Personalized Decentralized Federated Learning with Knowledge Distillation

  2023cites this paper
• Perada: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees

  2023cites this paper
• Federated Learning for Data Streams

  2023influential citation
• Fed-CO2: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learning

  2023influential citation
• Federated Deep Equilibrium Learning: Harnessing Compact Global Representations to Enhance Personalization

  2023cites this paper
• Factor-Assisted Federated Learning for Personalized Optimization with Heterogeneous Data

  2023cites this paper
• A Cautionary Tale: On the Role of Reference Data in Empirical Privacy Defenses

  2023cites this paper
• Evaluating and Enhancing the Robustness of Federated Learning System against Realistic Data Corruption

  2023cites this paper
• A Joint Training-Calibration Framework for Test-Time Personalization with Label Shift in Federated Learning

  2023cites this paper
• PAGE: Equilibrate Personalization and Generalization in Federated Learning

  2023cites this paper
• Federated Deep Equilibrium Learning: A Compact Shared Representation for Edge Communication Efficiency

  2023cites this paper
• Personalized Federated Learning with Inferred Collaboration Graphs

  2023influential citation
• Active Client Selection for Clustered Federated Learning

  2023cites this paper
• A Practical Recipe for Federated Learning Under Statistical Heterogeneity Experimental Design

  2023cites this paper