Large Language Model Supply Chain: A Research Agenda

The rapid advancement of large language models (LLMs) has revolutionized artificial intelligence, introducing unprecedented capabilities in natural language processing and multimodal content generation. However, the increasing complexity and scale of these models have given rise to a multifaceted supply chain that presents unique challenges across infrastructure, foundation models, and downstream applications. This article provides the first comprehensive research agenda of the LLM supply chain, offering a structured approach to identify critical challenges and opportunities through the dual lenses of software engineering (SE) and security and privacy (S&P). We begin by establishing a clear definition of the LLM supply chain, encompassing its components and dependencies. We then analyze each layer of the supply chain, presenting a vision for robust and secure LLM development, reviewing the current state of practices and technologies, and identifying key challenges and research opportunities. This work aims to bridge the existing research gap in systematically understanding the multifaceted issues within the LLM supply chain, offering valuable insights to guide future efforts in this rapidly evolving domain.

• Publication year

  2024

• Venue

  ACM Transactions on Software Engineering and Methodology

• Publication date

  2024-04-19

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.1145/3708531arXiv 2404.12736
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

  2024cited by this paper
• An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets

  2024cited by this paper
• What's documented in AI? Systematic Analysis of 32K AI Model Cards

  2024cited by this paper
• Towards the Self-Healing of Infrastructure as Code Projects Using Constrained LLM Technologies

  2024cited by this paper
• Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection

  2024cited by this paper
• EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models

  2024cited by this paper
• On the (In)Security of LLM App Stores

  2024cited by this paper
• Towards semantic versioning of open pre-trained language model releases on hugging face

  2024cited by this paper
• ReposVul: A Repository-Level High-Quality Vulnerability Dataset

  2024cited by this paper
• Large Language Model-Based Agents for Software Engineering: A Survey

  2024cited by this paper
• The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies

  2024cited by this paper
• Exploring Multi-Lingual Bias of Large Code Models in Code Generation

  2024cited by this paper
• Rethinking Interpretability in the Era of Large Language Models

  2024cited by this paper
• Towards Building a Robust Toxicity Predictor

  2024cited by this paper
• Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs

  2024cited by this paper
• Jailbreak Attacks and Defenses Against Large Language Models: A Survey

  2024cited by this paper
• CodeMirage: Hallucinations in Code Generated by Large Language Models

  2024cited by this paper
• Models Are Codes: Towards Measuring Malicious Code Poisoning Attacks on Pre-trained Model Hubs

  2024cited by this paper
• Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems

  2024cited by this paper
• Deep Learning Model Reuse in the HuggingFace Community: Challenges, Benefit and Trends

  2024cited by this paper
• Evading Data Contamination Detection for Language Models is (too) Easy

  2024cited by this paper
• Digger: Detecting Copyright Content Mis-usage in Large Language Model Training

  2024cited by this paper
• How do Hugging Face Models Document Datasets, Bias, and Licenses? An Empirical Study

  2024cited by this paper
• Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification

  2024cited by this paper
• The 2024 Foundation Model Transparency Index

  2024cited by this paper
• Model Merging and Safety Alignment: One Bad Model Spoils the Bunch

  2024cited by this paper
• Evaluating Large Language Models in Class-Level Code Generation

  2024cited by this paper
• LLM as a System Service on Mobile Devices

  2024cited by this paper
• A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems

  2024cited by this paper
• AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks

  2024cited by this paper
• Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices

  2024cited by this paper
• Concerned with Data Contamination? Assessing Countermeasures in Code Language Model

  2024influential reference
• LLM App Store Analysis: A Vision and Roadmap

  2024cited by this paper
• Systematic analysis of 32,111 AI model cards characterizes documentation practice in AI

  2024cited by this paper
• RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent

  2024cited by this paper
• Voices from the Frontier: A Comprehensive Analysis of the OpenAI Developer Forum

  2024cited by this paper
• SPICED: Syntactical Bug and Trojan Pattern Identification in A/MS Circuits using LLM-Enhanced Detection

  2024cited by this paper
• Agents in software engineering: survey, landscape, and vision

  2024cited by this paper
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

  2024cited by this paper
• JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs

  2024cited by this paper
• TrustLLM: Trustworthiness in Large Language Models

  2024influential reference
• A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models

  2024cited by this paper
• Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security

  2024cited by this paper
• Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence

  2024cited by this paper
• LeftoverLocals: Listening to LLM Responses Through Leaked GPU Local Memory

  2024cited by this paper
• Model Editing at Scale leads to Gradual and Catastrophic Forgetting

  2024cited by this paper
• Datasets for Large Language Models: A Comprehensive Survey

  2024cited by this paper
• Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs

  2024cited by this paper
• Trained without My Consent: Detecting Code Inclusion in Language Models Trained on Code

  2024cited by this paper
• LLM Factoscope: Uncovering LLMs' Factual Discernment through Measuring Inner States

  2024cited by this paper
• MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

  2024cited by this paper
• A Survey on Data Selection for Language Models

  2024cited by this paper
• Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models

  2024cited by this paper
• Challenges and Opportunities in Integrating LLMs into Continuous Integration/Continuous Deployment (CI/CD) Pipelines

  2024cited by this paper
• Harnessing the Power of General-Purpose LLMs in Hardware Trojan Design

  2024cited by this paper
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

  2024cited by this paper
• Generating Benchmarks for Factuality Evaluation of Language Models

  2023cited by this paper
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback

  2023cited by this paper
• Honeycomb: Secure and Efficient GPU Executions via Static Validation

  2023cited by this paper
• CodeScore: Evaluating Code Generation by Learning Code Execution

  2023cited by this paper
• Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias

  2023cited by this paper
• BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset

  2023cited by this paper
• Unlearning Bias in Language Models by Partitioning Gradients

  2023cited by this paper
• LLaMA: Open and Efficient Foundation Language Models

  2023cited by this paper
• An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry

  2023cited by this paper
• A Survey on Evaluation of Large Language Models

  2023cited by this paper
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model

  2023cited by this paper
• Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation

  2023cited by this paper
• Prompt Injection attack against LLM-integrated Applications

  2023cited by this paper
• Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents

  2023cited by this paper
• Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study

  2023cited by this paper
• What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks

  2023cited by this paper
• Visual Adversarial Examples Jailbreak Aligned Large Language Models

  2023cited by this paper
• RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems

  2023cited by this paper
• LLM-Assisted Code Cleaning For Training Accurate Code Generators

  2023cited by this paper
• Biases in Large Language Models: Origins, Inventory, and Discussion

  2023cited by this paper
• Extracting Training Data from Diffusion Models

  2023cited by this paper
• Flames: Benchmarking Value Alignment of Chinese Large Language Models

  2023cited by this paper
• Hot Pixels: Frequency, Power, and Temperature Attacks on GPUs and ARM SoCs

  2023cited by this paper
• Analyzing Leakage of Personally Identifiable Information in Language Models

  2023cited by this paper
• Assessing the Vulnerabilities of the Open-Source Artificial Intelligence (AI) Landscape: A Large-Scale Analysis of the Hugging Face Platform

  2023cited by this paper
• Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning

  2023cited by this paper
• Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

  2023cited by this paper
• The Foundation Model Transparency Index

  2023cited by this paper
• Similarity of Neural Network Models: A Survey of Functional and Representational Measures

  2023cited by this paper
• A Comprehensive Survey on Vector Database: Storage and Retrieval Technique, Challenge

  2023cited by this paper
• The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI

  2023cited by this paper
• Empirical Analysis of Vulnerabilities Life Cycle in Golang Ecosystem

  2023cited by this paper
• MalWuKong: Towards Fast, Accurate, and Multilingual Detection of Malicious Code Poisoning in OSS Supply Chains

  2023cited by this paper
• Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models

  2023cited by this paper
• Misusing Tools in Large Language Models With Visual Adversarial Examples

  2023cited by this paper
• A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation

  2023cited by this paper
• Proving Test Set Contamination in Black Box Language Models

  2023cited by this paper
• Spear Phishing With Large Language Models

  2023cited by this paper
• Adversarial Demonstration Attacks on Large Language Models

  2023cited by this paper
• Towards Mitigating Hallucination in Large Language Models via Self-Reflection

  2023cited by this paper
• MART: Improving LLM Safety with Multi-round Automatic Red-Teaming

  2023cited by this paper
• An Empirical Study of Malicious Code In PyPI Ecosystem

  2023cited by this paper
• BOMs Away! Inside the Minds of Stakeholders: A Comprehensive Study of Bills of Materials for Software Systems

  2023cited by this paper
• Large Language Model Alignment: A Survey

  2023cited by this paper

• Agentic AI as a Cybersecurity Attack Surface: Threats, Exploits, and Defenses in Runtime Supply Chains

  2026cites this paper
• Leveraging large language models to enhance multi-agent risk assessment in supply chain networks

  2026cites this paper
• Hidden Licensing Risks in the LLMware Ecosystem

  2026influential citation
• Conversational AI Interface Transforming Complex Supply Chain Pricing Data into Actionable Insights for Warehouse Management

  2026cites this paper
• Intelligence by design: Large language model work integration as strategic enablers for supply chain regeneration through digital and cognitive capabilities

  2026cites this paper
• Design and implementation of a local LLM-driven decision support system for AI-enhanced supply chain management

  2026cites this paper
• Learning to Generate Secure Code via Token-Level Rewards

  2026cites this paper
• An Empirical Study of Bugs in Modern LLM Agent Frameworks

  2026cites this paper
• Are cloud providers exploiting open-source? An exploratory study of Redis license change

  2026cites this paper
• Law LLM unlearning via interfere prompt, review output and update parameter: new challenges, method and baseline

  2025cites this paper
• Understanding the Supply Chain and Risks of Large Language Model Applications

  2025cites this paper
• Digital Divide (2.0): the Shadow of AI Technology

  2025cites this paper
• HuggingGraph: Understanding the Supply Chain of LLM Ecosystem

  2025influential citation
• CRMAgent: A Multi-Agent LLM System for E-Commerce CRM Message Template Generation

  2025cites this paper
• Defining ethically sourced code generation

  2025cites this paper
• On the synchronization between Hugging Face pre-trained language models and their upstream GitHub repository

  2025cites this paper
• NeMo: A Neuron-Level Modularizing-While-Training Approach for Decomposing DNN Models

  2025cites this paper
• AI-Enabled Customised Workflows for Smarter Supply Chain Optimisation: A Feasibility Study

  2025cites this paper
• Vulnerability Detection in Large Language Models: Addressing Security Concerns

  2025cites this paper
• SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV

  2025cites this paper
• Position: Privacy Is Not Just Memorization!

  2025cites this paper
• Research Directions in Software Supply Chain Security

  2025cites this paper
• SoK: Understanding Vulnerabilities in the Large Language Model Supply Chain

  2025cites this paper
• Towards Reliable Vector Database Management Systems: A Software Testing Roadmap for 2030

  2025cites this paper
• Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents

  2025cites this paper
• Q-FAKER: Query-free Hard Black-box Attack via Controlled Generation

  2025cites this paper
• Understanding Large Language Model Supply Chain: Structure, Domain, and Vulnerabilities

  2025cites this paper
• The Importance of AI Data Governance in Large Language Models

  2025cites this paper
• Toward Understanding Bugs in Vector Database Management Systems

  2025cites this paper
• Optimizing Large Language Models in Distributed Environments: A Holistic Approach to Efficiency, Ethics, and Governance

  2025cites this paper
• Large language models: applications, limitations, challenges, and recommendations in cybersecurity, digital forensics, and ethical hacking

  2025cites this paper
• Cyber Threat Intelligence for Hybrid Attacks: Leveraging LLMs and Data Spaces

  2025cites this paper
• A Novel Machine Learning-Optimized Framework for Systematic Analysis of Foundation Models in Healthcare: Comprehensive Algorithm Optimization With Governance-Driven Predictive Modeling

  2025cites this paper
• Skyline operators in multi-criteria decision making: A review of characterization, comparison, and perspectives

  2025cites this paper
• What We Know about AIBOMs: Results from a Multivocal Literature Review on Artificial Intelligence Bill of Materials

  2025cites this paper
• Large language models in the software supply chain: challenges and opportunities

  2025cites this paper
• A Characterization Study of Bugs in LLM Agent Workflow Orchestration Frameworks

  2025cites this paper
• Galene: A Toolkit for Encapsulating Software Engineering Practices in Marine Science AI Applications

  2025cites this paper
• Testing and Debugging Quantum Programs: The Road to 2030

  2024cites this paper
• Ecosystem of Large Language Models for Code

  2024cites this paper
• Large Language Model Supply Chain: Open Problems From the Security Perspective

  2024cites this paper
• Lifting the Veil on Composition, Risks, and Mitigations of the Large Language Model Supply Chain

  2024cites this paper
• Models Are Codes: Towards Measuring Malicious Code Poisoning Attacks on Pre-trained Model Hubs

  2024cites this paper
• LLMs for Enhanced Agricultural Meteorological Recommendations

  2024cites this paper
• LLM Supply Chain Provenance: A Blockchain-based Approach

  2024cites this paper
• On the (In)Security of LLM App Stores

  2024cites this paper
• GPTZoo: A Large-scale Dataset of GPTs for the Research Community

  2024cites this paper