Self-supervised Distillation Method for Lightweight Convolutional Networks

In deep learning visual tasks, convolutional neural networks (CNNs) have demonstrated excellent performance in various image classification scenarios. However, their large number of parameters and dependence on large-scale annotated data severely restrict the application of the model in small samples and resource-limited environments. To address this issue, we present a lightweight image classification framework that couples self-supervised teacher pre-training with knowledge distillation to a compact student. The teacher (ResNet-18) is pre-trained with SimCLR and, under the same unlabeled budget, with MoCo v2 and BYOL; the student (LiteNet-DSC) uses depthwise-separable convolutions with batch normalization. Beyond final-logit distillation, we add a lightweight feature-level transfer aligning two intermediate stages, and incorporate consistency regularization with semi-supervised pseudo-labels. On MNIST and CIFAR-10 under a low-label regime (CIFAR-10: 10k labeled / 35k unlabeled), the student reaches up to 93.1% top-1 with ~0.5M parameters and 0.50 ms single-image latency, about 5× faster than the teacher. Ablations show consistent gains from alternative SSL (BYOL ≥ MoCo v2 > SimCLR) and from feature-level transfer. The approach achieves a high-accuracy, low-overhead balance suitable for mobile and embedded deployment.

• Publication year

  2025

• Venue

  2025 6th International Conference on Computer Engineering and Intelligent Control (ICCEIC)

• Publication date

  2025-10-17

• Fields of study

  Not labeled

• Identifiers
  DOI 10.1109/ICCEIC67916.2025.11309140
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Noisy Label Processing for Classification: A Survey

  2024cited by this paper
• Pseudo Labeling Methods for Semi-Supervised Semantic Segmentation: A Review and Future Perspectives

  2024cited by this paper
• Falcon: lightweight and accurate convolution based on depthwise separable convolution

  2023cited by this paper
• Comparative Analysis for Slope Stability by Using Machine Learning Methods

  2023cited by this paper
• When Object Detection Meets Knowledge Distillation: A Survey

  2023cited by this paper
• Temporal-channel convolution with self-attention network for human activity recognition using wearable sensors

  2023cited by this paper
• SCConv: Spatial and Channel Reconstruction Convolution for Feature Redundancy

  2023cited by this paper
• Self-supervised learning advanced plant disease image classification with SimCLR

  2023cited by this paper
• MixUp based Cross-Consistency Training for Named Entity Recognition

  2022cited by this paper
• A Framework for Motivating Teacher-Student Relationships

  2022cited by this paper
• Review on Convolutional Neural Networks (CNN) in vegetation remote sensing

  2021cited by this paper
• Graph Self-Supervised Learning: A Survey

  2021cited by this paper
• A Comprehensive Survey of Loss Functions in Machine Learning

  2020cited by this paper
• Review of research on lightweight convolutional neural networks

  2020cited by this paper
• A survey of image classification methods and techniques

  2014cited by this paper

• No citing papers are available for this paper.