Efficient Federated Fine-Tuning via Zeroth-Order Optimization for Resource-Constrained Edge Devices

Federated fine-tuning of large language models (LLMs) is critical for enabling privacy-preserving learning on edge devices. Traditional backpropagation, however, imposes high memory and computational demands, making it impractical for deployment on lightweight hardware such as Neural Processing Units (NPUs). This paper presents a simulation-based analysis comparing three optimization strategies: standard backpropagation-based Federated Averaging (FedAvg), zerothorder (ZO) full-rank, and ZO low-rank estimators. A compact transformer classifier is trained on a synthetic AGNEWS-like dataset to evaluate convergence behavior, rank sensitivity, and scalability across federated clients. Experimental results demonstrate that the ZO low-rank method achieves smooth and stable convergence, delivering accuracy comparable to backpropagation while significantly reducing memory overhead. These findings highlight forward-only, low-rank ZO optimization as an effective and backpropagation-free alternative for federated fine-tuning of LLMs on NPU-constrained environments.

• Publication year

  2025

• Venue

  International Conference on Computational Intelligence and Communication Networks

• Publication date

  2025-12-20

• Fields of study

  Not labeled

• Identifiers
  DOI 10.1109/CICN67655.2025.11368000
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• A Survey on Efficient Large Language Model Training: From Data-centric Perspectives

  2025cited by this paper
• Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scale

  2025cited by this paper
• Federated learning: Overview, strategies, applications, tools and future directions

  2024cited by this paper
• Federated Learning-Based Architecture for Personalized Next Emoji Prediction for Social Media Comments

  2024cited by this paper
• Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models

  2024cited by this paper
• FwdLLM: Efficient FedLLM using Forward Gradient

  2023cited by this paper
• The Power of Scale for Parameter-Efficient Prompt Tuning

  2021cited by this paper
• LoRA: Low-Rank Adaptation of Large Language Models

  2021cited by this paper
• Prefix-Tuning: Optimizing Continuous Prompts for Generation

  2021cited by this paper
• Communication-efficient Decentralized Machine Learning over Heterogeneous Networks

  2020influential reference
• Parameter-Efficient Transfer Learning for NLP

  2019cited by this paper
• Simple random search of static linear policies is competitive for reinforcement learning

  2018cited by this paper
• ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models

  2017cited by this paper
• Federated Multi-Task Learning

  2017cited by this paper
• Memory-Efficient Backpropagation Through Time

  2016cited by this paper
• Training Deep Nets with Sublinear Memory Cost

  2016cited by this paper
• Orthogonal Random Features

  2016cited by this paper
• Random Gradient-Free Minimization of Convex Functions

  2015cited by this paper
• An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedback

  2015cited by this paper
• A MADS Algorithm with a Progressive Barrier for Derivative-Free Nonlinear Programming

  2007cited by this paper

• No citing papers are available for this paper.