When Object Detection Meets Knowledge Distillation: A Survey

Object detection (OD) is a crucial computer vision task that has seen the development of many algorithms and models over the years. While the performance of current OD models has improved, they have also become more complex, making them impractical for industry applications due to their large parameter size. To tackle this problem, knowledge distillation (KD) technology was proposed in 2015 for image classification and subsequently extended to other visual tasks due to its ability to transfer knowledge learned by complex teacher models to lightweight student models. This paper presents a comprehensive survey of KD-based OD models developed in recent years, with the aim of providing researchers with an overview of recent progress in the field. We conduct an in-depth analysis of existing works, highlighting their advantages and limitations, and explore future research directions to inspire the design of models for related tasks. We summarize the basic principles of designing KD-based OD models, describe related KD-based OD tasks, including performance improvements for lightweight models, catastrophic forgetting in incremental OD, small object detection, and weakly/semi-supervised OD. We also analyze novel distillation techniques, i.e. different types of distillation loss, feature interaction between teacher and student models, etc. Additionally, we provide an overview of the extended applications of KD-based OD models on specific datasets, such as remote sensing images and 3D point cloud datasets. We compare and analyze the performance of different models on several common datasets and discuss promising directions for solving specific OD problems.

• Publication year

  2023

• Venue

  IEEE Transactions on Pattern Analysis and Machine Intelligence

• Publication date

  2023-03-15

• Fields of study

  Medicine, Computer Science

• Identifiers
  DOI 10.1109/TPAMI.2023.3257546PMID 37028387
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Multilevel Attention-Based Sample Correlations for Knowledge Distillation

  2023cited by this paper
• RD-IOD: Two-Level Residual-Distillation-Based Triple-Network for Incremental Object Detection

  2022cited by this paper
• Collaborative Knowledge Distillation via Multiknowledge Transfer

  2022cited by this paper
• Hierarchical Multi-Attention Transfer for Knowledge Distillation

  2022cited by this paper
• IDa-Det: An Information Discrepancy-aware Distillation for 1-bit Detectors

  2022cited by this paper
• Few-Shot Object Detection Based on Self-Knowledge Distillation

  2022cited by this paper
• YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications

  2022cited by this paper
• Detecting Vehicles on the Edge: Knowledge Distillation to Improve Performance in Heterogeneous Road Traffic

  2022cited by this paper
• Distillation Using Oracle Queries for Transformer-based Human-Object Interaction Detection

  2022cited by this paper
• Single-Domain Generalized Object Detection in Urban Scene via Cyclic-Disentangled Self-Distillation

  2022cited by this paper
• Weakly Supervised Part-Based Method for Combined Object Detection in Remote Sensing Imagery

  2022cited by this paper
• Boosting 3D Object Detection by Simulating Multimodality on Point Clouds

  2022cited by this paper
• Extracting and Distilling Direction-Adaptive Knowledge for Lightweight Object Detection in Remote Sensing Images

  2022cited by this paper
• itKD: Interchange Transfer-based Knowledge Distillation for 3D Object Detection

  2022cited by this paper
• Towards Efficient 3D Object Detection with Knowledge Distillation

  2022cited by this paper
• Balanced knowledge distillation for one-stage object detector

  2022cited by this paper
• Semi-supervised Knowledge Distillation for Tiny Defect Detection

  2022cited by this paper
• Adaptive Knowledge Distillation for Lightweight Remote Sensing Object Detectors Optimizing

  2022cited by this paper
• Incremental-DETR: Incremental Few-Shot Object Detection via Self-Supervised Learning

  2022cited by this paper
• Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism

  2022cited by this paper
• Cross Domain Object Detection by Target-Perceived Dual Branch Distillation

  2022cited by this paper
• End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation

  2022cited by this paper
• Re-examining Distillation For Continual Object Detection

  2022cited by this paper
• DD-YOLO: An object detection method combining knowledge distillation and Differentiable Architecture Search

  2022cited by this paper
• Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data

  2022cited by this paper
• Instance Relation Graph Guided Source-Free Domain Adaptive Object Detection

  2022cited by this paper
• LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection

  2022influential reference
• Group channel pruning and spatial attention distilling for object detection

  2022cited by this paper
• Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillation

  2022cited by this paper
• Adaptive Perspective Distillation for Semantic Segmentation

  2022cited by this paper
• Decoupled Knowledge Distillation

  2022cited by this paper
• Prediction-Guided Distillation for Dense Object Detection

  2022cited by this paper
• Semantic Distillation Guided Salient Object Detection

  2022influential reference
• Multi-level knowledge distillation for low-resolution object detection and facial expression recognition

  2022cited by this paper
• On-Device Saliency Prediction Based on Pseudoknowledge Distillation

  2022cited by this paper
• A Light-Weight CNN for Object Detection with Sparse Model and Knowledge Distillation

  2022cited by this paper
• Dynamic Rectification Knowledge Distillation

  2022cited by this paper
• Adaptive Instance Distillation for Object Detection in Autonomous Driving

  2022cited by this paper
• MonoDistill: Learning Spatial Features for Monocular 3D Object Detection

  2022influential reference
• Localization Distillation for Object Detection

  2022cited by this paper
• Enhancing Mid–Low-Resolution Ship Detection With High-Resolution Feature Distillation

  2022cited by this paper
• Lipschitz Continuity Guided Knowledge Distillation

  2021cited by this paper
• CADN: A weakly supervised learning-based category-aware object detection network for surface defect detection

  2021cited by this paper
• Data-free Knowledge Distillation for Object Detection

  2021cited by this paper
• Learning Slimming SAR Ship Object Detector Through Network Pruning and Knowledge Distillation

  2021cited by this paper
• A compression pipeline for one-stage object detection model

  2021cited by this paper
• There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledge

  2021influential reference
• General Instance Distillation for Object Detection

  2021cited by this paper
• In Defense of Knowledge Distillation for Task Incremental Learning and Its Application in 3D Object Detection

  2021cited by this paper
• ZoomInNet: A Novel Small Object Detector in Drone Images with Cross-Scale Knowledge Distillation

  2021cited by this paper
• Distilling Knowledge via Knowledge Review

  2021cited by this paper
• Revisiting Knowledge Distillation for Object Detection

  2021cited by this paper
• Multi-Scale Aligned Distillation for Low-Resolution Detection

  2021influential reference
• Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors

  2021cited by this paper
• Towards Black-Box Explainability with Gaussian Discriminant Knowledge Distillation

  2021cited by this paper
• Multi-View Correlation Distillation for Incremental Object Detection

  2021cited by this paper
• Object-Oriented Relational Distillation for Object Detection

  2021cited by this paper
• Attention-Guided Knowledge Distillation for Efficient Single-Stage Detector

  2021influential reference
• Hierarchical Self-supervised Augmented Knowledge Distillation

  2021cited by this paper
• Object detection based on self feature distillation

  2021influential reference
• Revisiting knowledge distillation for light-weight visual object detection

  2021cited by this paper
• G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitation

  2021influential reference
• Improving Object Detection by Label Assignment Distillation

  2021influential reference
• Knowledge Distillation for Object Detection Based on Mutual Information

  2021influential reference
• Research on Object Detection Network Based on Knowledge Distillation

  2021influential reference
• LGD: Label-guided Self-distillation for Object Detection

  2021influential reference
• One-stage object detection knowledge distillation via adversarial learning

  2021cited by this paper
• Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

  2021cited by this paper
• Boosting End-to-end Multi-Object Tracking and Person Search via Knowledge Distillation

  2021cited by this paper
• Instance-Conditional Knowledge Distillation for Object Detection

  2021cited by this paper
• Response-based Distillation for Incremental Object Detection

  2021influential reference
• Active Learning for Lane Detection: A Knowledge Distillation Approach

  2021cited by this paper
• Hybrid Deep Learning Vision-based Models for Human Object Interaction Detection by Knowledge Distillation

  2021cited by this paper
• Focal and Global Knowledge Distillation for Detectors

  2021cited by this paper
• Photon-Limited Object Detection using Non-local Feature Matching and Knowledge Distillation

  2021cited by this paper
• Knowledge Distillation for Low-Power Object Detection: A Simple Technique and Its Extensions for Training Compact Models Using Unlabeled Data

  2021cited by this paper
• Spatial likelihood voting with self-knowledge distillation for weakly supervised object detection

  2021influential reference
• Localization Distillation for Dense Object Detection

  2021cited by this paper
• An Empirical Study of Adder Neural Networks for Object Detection

  2021cited by this paper
• Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-guided Feature Imitation

  2021cited by this paper
• Fast Video Salient Object Detection via Spatiotemporal Knowledge Distillation

  2020cited by this paper
• Continual Universal Object Detection

  2020influential reference
• Extended Feature Pyramid Network for Small Object Detection

  2020cited by this paper
• Incremental Object Detection via Meta-Learning

  2020cited by this paper
• Squeezed Deep 6DoF Object Detection using Knowledge Distillation

  2020cited by this paper
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks

  2020cited by this paper
• Distilling Knowledge from Refinement in Multiple Instance Detection Networks

  2020cited by this paper
• Knowledge Distillation: A Survey

  2020cited by this paper
• A2dele: Adaptive and Attentive Depth Distiller for Efficient RGB-D Salient Object Detection

  2020cited by this paper
• Online Knowledge Distillation via Collaborative Learning

  2020cited by this paper
• Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation

  2020influential reference
• Two-Level Residual Distillation based Triple Network for Incremental Object Detection

  2020cited by this paper
• Weakly Supervised 3D Object Detection from Point Clouds

  2020influential reference
• Prime-Aware Adaptive Distillation

  2020cited by this paper
• Mobile Centernet for Embedded Deep Learning Object Detection

  2020cited by this paper
• Real Time Object Detection for Traffic Based on Knowledge Distillation: 3rd Place Solution to Pair Competition

  2020cited by this paper
• Lifelong Object Detection

  2020cited by this paper
• KNOWLEDGE DISTILLATION USING GANS FOR FAST OBJECT DETECTION

  2020cited by this paper
• Multi-Frame to Single-Frame: Knowledge Distillation for 3D Object Detection

  2020cited by this paper
• Compact Cloud Detection with Bidirectional Self-Attention Knowledge Distillation

  2020influential reference

• Knowledge-Centric Split Federated Learning With Partial Client Participation for Non-IID Data

  2026cites this paper
• StegaFFD: Privacy-Preserving Face Forgery Detection via Fine-Grained Steganographic Domain Lifting

  2026cites this paper
• A Novel Knowledge Distillation Framework to Optimize DETR for Sonar Image Object Detection

  2026cites this paper
• Research on Lightweight Rose Disease Detection Based on Transferable Feature Representation.

  2026cites this paper
• Quality over quantity: a data-centric survey of annotation errors in object detection datasets

  2026cites this paper
• A botnet detection method for encrypted DNS traffic based on multi-branch knowledge distillation

  2026cites this paper
• Efficient human pose estimation in complex coal mining scenes via Keypoint Partitioning Adaptive Convolution

  2026cites this paper
• Adaptive tucker decomposition-based progressive model compression for convolutional neural networks

  2026cites this paper
• CDA-Net: Cross dimensional attention network for wetland bird detection

  2026cites this paper
• Integrating Knowledge Distillation into AlphaGo Zero for Gomoku Board Game

  2025cites this paper
• Enhancing the generalization capability of 2D array pointer networks through multiple teacher-forcing knowledge distillation

  2025cites this paper
• SAMKD: Spatial-aware Adaptive Masking Knowledge Distillation for Object Detection

  2025cites this paper
• A comprehensive scoping review on machine learning-based fetal echocardiography analysis

  2025cites this paper
• Advancing Trans-Domain Classification With Knowledge Distillation: Bridging LIDAR and Image Data

  2025cites this paper
• DCKD: Distribution-Corrected Knowledge Distillation for Enhanced Industrial Defect Detection

  2025cites this paper
• Hardware-Aware DNN Compression for Homogeneous Edge Devices

  2025cites this paper
• Structural and Statistical Texture Knowledge Distillation and Learning for Segmentation

  2025cites this paper
• LT-DETR: lightweight UAV object detection and dual knowledge distillation for remote sensing scenarios

  2025cites this paper
• LGVLM-mIoT: A Lightweight Generative Visual-Language Model for Multilingual IoT Applications

  2025cites this paper
• Self-Distillation Attention for Efficient and Accurate Motion Prediction in Autonomous Driving

  2025cites this paper
• Facial Expression Recognition With Heatmap Neighbor Contrastive Learning

  2025cites this paper
• ReRAW: RGB-to-RAW Image Reconstruction via Stratified Sampling for Efficient Object Detection on the Edge

  2025cites this paper
• Adaptive generative knowledge distillation for dense object detection with reused detector

  2025cites this paper
• Neighborhood relation-based knowledge distillation for image classification

  2025cites this paper
• Condensation of Data and Knowledge for Network Traffic Classification: Techniques, Applications, and Open Issues

  2025cites this paper
• Survey on deep learning-based weakly supervised salient object detection

  2025cites this paper
• TKSF-KAN: Transformer-enhanced oat yield modeling and transferability across major oat-producing regions in China using UAV multisource data

  2025cites this paper
• IoT-Based Precision Litchi Tracking and Counting Method Using Gated Metrics

  2025cites this paper
• Boundary-Aware Semantic Bird-Eye-View Map Generation Based on Conditional Diffusion Models

  2025cites this paper
• Efficient vision-based vehicle speed estimation

  2025cites this paper
• PTAformer: transformer-based global-local feature interaction framework for behavior recognition

  2025cites this paper
• REFNet: reparameterized feature enhancement and fusion network for underwater blur target recognition

  2025cites this paper
• Self-supervised rotation-aware guided attentive distilled transformer network for crowd face detection

  2025cites this paper
• Gearbox fault diagnosis based on multi-channel orthogonal attention network

  2025cites this paper
• A Lightweight YOLOv8s Algorithm for Ceiling Fan Blade Defect Detection With Optimized Pruning and Knowledge Distillation

  2025cites this paper
• VPA: Multi-Modal Virtual Point Augmentation for 3D Object Detection

  2025cites this paper
• Evolution of ReID: From Early Methods to LLM Integration

  2025cites this paper
• Weighted cross-integrated fusion network based on knowledge distillation for multi-modal personality recognition

  2025cites this paper
• Investigating Synthetic-to-Real Transfer Robustness for Stereo Matching and Optical Flow Estimation

  2025cites this paper
• Frugal Object Detection Models: Solutions, Challenges and Future Directions

  2025cites this paper
• Improving knowledge distillation via multi-level normalization and multi-level decoupling

  2025cites this paper
• Dynamic subgraph pruning and causal-aware knowledge distillation for temporal knowledge graphs

  2025influential citation
• SCRED-Distillation: Improving Low-Dose CT Image Quality via Feature Fusion and Mutual Learning

  2025cites this paper
• Distilling Balanced Knowledge from a Biased Teacher

  2025cites this paper
• Lightweight crack detection in complex environments via model pruning and knowledge distillation

  2025cites this paper
• MST-Distill: Mixture of Specialized Teachers for Cross-Modal Knowledge Distillation

  2025cites this paper
• Few-Shot Learning in Video and 3D Object Detection: A Survey

  2025cites this paper
• Plastic Distillation and Local Class Augment for Federated Class Incremental Learning

  2025cites this paper
• Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Review

  2025cites this paper
• Optimizing Federated Learning With Aggregation Strategies: A Comprehensive Survey

  2025cites this paper
• A Survey on Transformer Architecture Design for Abnormal Behavior Recognition Tasks

  2025cites this paper
• End-to-End Learning for Monocular 3D Human Pose Estimation

  2025cites this paper
• Robust contrastive knowledge distillation for long-tailed noisy class labels

  2025cites this paper
• A Novel Compression Framework for YOLOv8: Achieving Real-Time Aerial Object Detection on Edge Devices via Structured Pruning and Channel-Wise Distillation

  2025cites this paper
• Knowledge Distillation for Single Pixel Imaging Classification

  2025cites this paper
• Back and Forward Incremental Learning Through Knowledge Distillation for Object Detection Unmanned Aerial Vehicles

  2025cites this paper
• Instance mask alignment for object detection knowledge distillation

  2025cites this paper
• LMFL-YOLO: a lightweight multi-scale fusion and localization-enhanced YOLO network for steel surface defect detection

  2025cites this paper
• Real-Time Vehicle Detection and Tracking Using Low-Cost Roadside FMCW LiDAR

  2025cites this paper
• Lightweight bearing fault diagnosis via decoupled distillation and low rank adaptation

  2025cites this paper
• DSAD: Multi-Directional Contrast Spatial Attention-Driven Feature Distillation for Infrared Small Target Detection

  2025influential citation
• Feature-aligned distillation for dense object detection via refined semantic guidance and distribution consistency

  2025cites this paper
• Query-Distill: Cloud-Fog collaboration with knowledge distillation and confidence-based querying for smart buildings

  2025cites this paper
• QRT-DETR: Post-training quantization for real-time detection transformer

  2025cites this paper
• Diffusion-based knowledge distillation for effective multi-organ segmentation with reduced computational time

  2025cites this paper
• Rotation-Invariant Knowledge Distillation for Remote Sensing Object Detection

  2025cites this paper
• Knowledge Integrity in Large Language Models: A State-of-The-Art Review

  2025cites this paper
• Recent Real-Time Aerial Object Detection Approaches, Performance, Optimization, and Efficient Design Trends for Onboard Performance: A Survey

  2025cites this paper
• YolovN-CBi: A Lightweight and Efficient Architecture for Real-Time Detection of Small UAVs

  2025cites this paper
• WIOC-RVFL: Eliminating objective biases by weighted integrated optimization method

  2025cites this paper
• Self-supervised Distillation Method for Lightweight Convolutional Networks

  2025cites this paper
• ActivityCLIP: Enhancing group activity recognition by mining complementary information from text to supplement image modality

  2025cites this paper
• An Efficient Deep Neural Network for Surface Defect Detection in Industrial Edge Sensing

  2025cites this paper
• Deep learning-based smoker classification and detection: An overview and evaluation

  2025cites this paper
• A Comprehensive Survey of Machine Learning Techniques and Models for Object Detection

  2025cites this paper
• DCMSTRD: End-to-end Dense Captioning via Multi-Scale Transformer Decoding

  2024cites this paper
• Monitoring-Based Traffic Participant Detection in Urban Mixed Traffic: A Novel Dataset and A Tailored Detector

  2024cites this paper
• Modular Learning: Agile Development of Robust Traffic Sign Recognition

  2024cites this paper
• Knowledge Distillation for Object Detection Based on Feature Enhancement and Dynamic Fusion

  2024cites this paper
• LIDL4Oliv: A Lightweight Incremental Deep Learning Model for Classifying Olive Diseases in Images

  2024cites this paper
• The Research on a Lightweight Intelligent Distributed Feeder Automation Model Based on Knowledge Distillation

  2024cites this paper
• Distributed Semantic Communications for Multimodal Audio-Visual Parsing Tasks

  2024cites this paper
• Distilling object detectors with efficient logit mimicking and mask-guided feature imitation

  2024cites this paper
• NAIR: An Efficient Distributed Deep Learning Architecture for Resource Constrained IoT System

  2024cites this paper
• CRKD: Enhanced Camera-Radar Object Detection with Cross-Modality Knowledge Distillation

  2024cites this paper
• Sign language detection using convolutional neural network

  2024cites this paper
• Neighborhood Multi-Compound Transformer for Point Cloud Registration

  2024cites this paper
• Advanced intelligent monitoring technologies for animals: A survey

  2024cites this paper
• Model compression of deep neural network architectures for visual pattern recognition: Current status and future directions

  2024cites this paper
• Robust feature knowledge distillation for enhanced performance of lightweight crack segmentation models

  2024cites this paper
• Federated Distillation: A Survey

  2024cites this paper
• CKD: Contrastive Knowledge Distillation From a Sample-Wise Perspective

  2024cites this paper
• End-Edge-Cloud Collaborative Computing for Deep Learning: A Comprehensive Survey

  2024cites this paper
• Heterogeneous Energy Harvesting Techniques for Smart Home IoT Acceleration

  2024cites this paper
• Air Traffic Flow Prediction with Spatiotemporal Knowledge Distillation Network

  2024cites this paper
• Active Object Detection with Knowledge Aggregation and Distillation from Large Models

  2024cites this paper
• Rethinking Early-Fusion Strategies for Improved Multispectral Object Detection

  2024cites this paper
• FATE: Learning Effective Binary Descriptors With Group Fairness

  2024cites this paper
• DKDL-Net: A Lightweight Bearing Fault Detection Model via Decoupled Knowledge Distillation and Low-Rank Adaptation Fine-tuning

  2024cites this paper
• An Efficient Segmentation Model With Multipath Attention Mechanism Enabling Particle Size Characterization of Coal Dust

  2024cites this paper