LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends

With the broader adoption and highly successful development of Large Language Models (LLMs), there has been growing interest and demand for applying LLMs to autonomous driving technology. Driven by their natural language understanding and reasoning capabilities, LLMs have the potential to enhance various aspects of autonomous driving systems, from perception and scene understanding to interactive decision-making. In this paper, we first introduce the novel concept of designing Large Language Models for Autonomous Driving (LLM4AD), followed by a review of existing LLM4AD studies. Then, we propose a comprehensive benchmark for evaluating the instruction-following and reasoning abilities of LLM4AD systems, which includes LaMPilot-Bench, CARLA Leaderboard 1.0 Benchmark in simulation and NuPlanQA for multi-view visual question answering. Furthermore, we conduct extensive real-world experiments on autonomous vehicle platforms, examining both on-cloud and on-edge LLM deployment for personalized decision-making and motion control. Next, we explore the future trends of integrating language diffusion models into autonomous driving, exemplified by the proposed ViLaD (Vision-Language Diffusion) framework. Finally, we discuss the main challenges of LLM4AD, including latency, deployment, security and privacy, safety, trust and transparency, and personalization.

• Publication year

  2024

• Venue

  Unknown venue

• Publication date

  2024-10-20

• Fields of study

  Computer Science, Engineering

• Identifiers
  arXiv 2410.15281
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• LLM-augmented hierarchical reinforcement learning for human-like decision-making of autonomous driving

  2025cited by this paper
• VDT-Auto: End-to-end Autonomous Driving with VLM-Guided Diffusion Transformers

  2025cited by this paper
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

  2025cited by this paper
• TeLL-Drive: Enhancing Autonomous Driving with Teacher LLM-Guided Deep Reinforcement Learning

  2025cited by this paper
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning

  2025cited by this paper
• A Hierarchical Test Platform for Vision Language Model (VLM)-Integrated Real-World Autonomous Driving

  2025cited by this paper
• MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Cases

  2025cited by this paper
• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning

  2025cited by this paper
• Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Driving

  2025cited by this paper
• ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving

  2025cited by this paper
• dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching

  2025cited by this paper
• Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding

  2025cited by this paper
• ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Driving

  2025cited by this paper
• LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning

  2025cited by this paper
• AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Driving

  2025cited by this paper
• Towards Human-Centric Autonomous Driving: A Fast-Slow Architecture Integrating Large Language Model Guidance with Reinforcement Learning

  2025cited by this paper
• DriveAgent-R1: Advancing VLM-based Autonomous Driving with Hybrid Thinking and Active Perception

  2025cited by this paper
• ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation

  2025cited by this paper
• AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Driving

  2025cited by this paper
• NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models

  2025influential reference
• AutoDrive-R$^2$: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Driving

  2025cited by this paper
• Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving

  2024cited by this paper
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

  2024cited by this paper
• VLP: Vision Language Planning for Autonomous Driving

  2024influential reference
• Human-centric Reward Optimization for Reinforcement Learning-based Automated Driving using Large Language Models

  2024cited by this paper
• RoboCar: A Rapidly Deployable Open Source Platform for Autonomous Driving Research

  2024cited by this paper
• How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites

  2024cited by this paper
• Developing an Automated Vehicle Research Platform by Integrating Autoware with the DataSpeed Drive-By-Wire System

  2024cited by this paper
• Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Driving

  2024cited by this paper
• Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving

  2024cited by this paper
• DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving

  2024cited by this paper
• Unifying F1TENTH Autonomous Racing: Survey, Methods and Benchmarks

  2024cited by this paper
• DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models

  2024cited by this paper
• RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model

  2024influential reference
• Driving Everywhere with Large Language Model Policy Adaptation

  2024cited by this paper
• LangProp: A code optimization framework using Language Models applied to driving

  2024cited by this paper
• One Model to Drift Them All: Physics-Informed Conditional Diffusion Model for Driving at the Limits

  2024cited by this paper
• OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving

  2024influential reference
• MWFormer: Multi-Weather Image Restoration Using Degradation-Aware Transformers

  2024cited by this paper
• Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs

  2024cited by this paper
• SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models

  2024cited by this paper
• On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validation

  2024cited by this paper
• Learning Autonomous Driving Tasks via Human Feedbacks with Large Language Models

  2024cited by this paper
• EMMA: End-to-End Multimodal Model for Autonomous Driving

  2024influential reference
• Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving

  2024cited by this paper
• Scaling up Masked Diffusion Models on Text

  2024cited by this paper
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

  2024influential reference
• LLM Pruning and Distillation in Practice: The Minitron Approach

  2024cited by this paper
• REDFormer: Radar Enlightens the Darkness of Camera Perception With Transformers

  2024cited by this paper
• LLaVA-OneVision: Easy Visual Task Transfer

  2024cited by this paper
• The Llama 3 Herd of Models

  2024cited by this paper
• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking

  2024cited by this paper
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

  2024cited by this paper
• Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHF

  2024cited by this paper
• Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving

  2024cited by this paper
• Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

  2023cited by this paper
• Mixed-Integer Motion Planning on German Roads Within the Apollo Driving Stack

  2023cited by this paper
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

  2023influential reference
• A Survey of Large Language Models

  2023cited by this paper
• WOMD-LiDAR: Raw Sensor Dataset Benchmark for Motion Forecasting

  2023influential reference
• Visual Instruction Tuning

  2023influential reference
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

  2023cited by this paper
• PaLM 2 Technical Report

  2023cited by this paper
• QLoRA: Efficient Finetuning of Quantized LLMs

  2023cited by this paper
• NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario

  2023cited by this paper
• Voyager: An Open-Ended Embodied Agent with Large Language Models

  2023cited by this paper
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day

  2023cited by this paper
• Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

  2023influential reference
• Imitation with Spatial-Temporal Heatmap: 2nd Place Solution for NuPlan Challenge

  2023cited by this paper
• End-to-End Autonomous Driving: Challenges and Frontiers

  2023cited by this paper
• DriveIRL: Drive in Real Life with Inverse Reinforcement Learning

  2023cited by this paper
• LLM Adaptive PID Control for B5G Truck Platooning Systems

  2023cited by this paper
• Drive Like a Human: Rethinking Autonomous Driving with Large Language Models

  2023cited by this paper
• Llama 2: Open Foundation and Fine-Tuned Chat Models

  2023influential reference
• (Ab)using Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs

  2023cited by this paper
• End-to-End Multimodal Sensor Dataset Collection Framework for Autonomous Vehicles

  2023cited by this paper
• A survey on large language model based autonomous agents

  2023cited by this paper
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

  2023cited by this paper
• HiLM-D: Enhancing MLLMs with Multi-scale High-Resolution Details for Autonomous Driving

  2023cited by this paper
• Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning

  2023cited by this paper
• Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles

  2023influential reference
• The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"

  2023cited by this paper
• Qwen Technical Report

  2023cited by this paper
• DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models

  2023influential reference
• EDGAR: An Autonomous Driving Research Platform - From Feature Development to Real-World Application

  2023cited by this paper
• DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model

  2023cited by this paper
• GPT-Driver: Learning to Drive with GPT

  2023cited by this paper
• Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving

  2023cited by this paper
• LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving

  2023influential reference
• Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models

  2023cited by this paper
• A Language Agent for Autonomous Driving

  2023cited by this paper
• Dolphins: Multimodal Language Model for Driving

  2023cited by this paper
• LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs

  2023influential reference
• LMDrive: Closed-Loop End-to-End Driving with Large Language Models

  2023cited by this paper
• DriveMLM: aligning multi-modal large language models with behavioral planning states for autonomous driving

  2023cited by this paper
• Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

  2023cited by this paper
• DriveLM: Driving with Graph Visual Question Answering

  2023influential reference
• LingoQA: Visual Question Answering for Autonomous Driving

  2023cited by this paper
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

  2023cited by this paper
• Large Language Models for Autonomous Driving: Real-World Experiments

  2023cited by this paper

• LLM and AI Agents for Autonomous Systems: A Survey of Applications, Datasets, and Security Challenges

  2026cites this paper
• Efficient and Explainable End-to-End Autonomous Driving via Masked Vision-Language-Action Diffusion

  2026cites this paper