ResNet Sparsifier: Learning Strict Identity Mappings in Deep Residual Networks

A family of super deep networks, referred to as residual networks or ResNet, achieved record-beating performance in various visual tasks such as image recognition, object detection, and semantic segmentation. The ability to train very deep networks naturally pushed the researchers to use enormous resources to achieve the best performance. Consequently, in many applications super deep residual networks were employed for just a marginal improvement in performance. In this paper, we propose epsilon-ResNet that allows us to automatically discard redundant layers, which produces responses that are smaller than a threshold epsilon, with a marginal or no loss in performance. The epsilon-ResNet architecture can be achieved using a few additional rectified linear units in the original ResNet. Our method does not use any additional variables nor numerous trials like other hyper-parameter optimization techniques. The layer selection is achieved using a single training process and the evaluation is performed on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets. In some instances, we achieve about 80% reduction in the number of parameters.

• Publication year

  2018

• Venue

  Unknown venue

• Publication date

  2018-04-05

• Fields of study

  Computer Science

• Identifiers
  arXiv 1804.01661
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• SphereFace: Deep Hypersphere Embedding for Face Recognition

  2017cited by this paper
• Convolutional Networks with Adaptive Computation Graphs

  2017cited by this paper
• CASENet: Deep Category-Aware Semantic Edge Detection

  2017cited by this paper
• Bounding and Counting Linear Regions of Deep Neural Networks

  2017cited by this paper
• Data-Driven Sparse Structure Selection for Deep Neural Networks

  2017cited by this paper
• Large-Scale Evolution of Image Classifiers

  2017cited by this paper
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

  2017cited by this paper
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

  2016cited by this paper
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks

  2016cited by this paper
• Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

  2016cited by this paper
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

  2016cited by this paper
• Pyramid Scene Parsing Network

  2016cited by this paper
• Identity Mappings in Deep Residual Networks

  2016influential reference
• R-FCN: Object Detection via Region-based Fully Convolutional Networks

  2016cited by this paper
• Wider or Deeper: Revisiting the ResNet Model for Visual Recognition

  2016cited by this paper
• Aggregated Residual Transformations for Deep Neural Networks

  2016cited by this paper
• Learning the Number of Neurons in Deep Networks

  2016cited by this paper
• Neural Architecture Search with Reinforcement Learning

  2016cited by this paper
• Pruning Filters for Efficient ConvNets

  2016cited by this paper
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size

  2016cited by this paper
• Less Is More: Towards Compact CNNs

  2016cited by this paper
• Pruning Convolutional Neural Networks for Resource Efficient Inference

  2016cited by this paper
• Highway Networks

  2015cited by this paper
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

  2015cited by this paper
• Deep Residual Learning for Image Recognition

  2015cited by this paper
• Quantized Convolutional Neural Networks for Mobile Devices

  2015cited by this paper
• Holistically-Nested Edge Detection

  2015cited by this paper
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

  2015cited by this paper
• Object Detection Networks on Convolutional Feature Maps

  2015cited by this paper
• Cyclical Learning Rates for Training Neural Networks

  2015cited by this paper
• Learning both Weights and Connections for Efficient Neural Network

  2015cited by this paper
• Sparse Convolutional Neural Networks

  2015cited by this paper
• Efficient and Robust Automated Machine Learning

  2015cited by this paper
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

  2014cited by this paper
• Speeding up Convolutional Neural Networks with Low Rank Expansions

  2014cited by this paper
• Efficient and accurate approximations of nonlinear convolutional networks

  2014cited by this paper
• Going deeper with convolutions

  2014cited by this paper
• ImageNet Large Scale Visual Recognition Challenge

  2014cited by this paper
• On the Number of Linear Regions of Deep Neural Networks

  2014cited by this paper
• On the importance of initialization and momentum in deep learning

  2013cited by this paper
• Some Improvements on Deep Convolutional Neural Network Based Image Classification

  2013cited by this paper
• Practical Bayesian Optimization of Machine Learning Algorithms

  2012cited by this paper
• Efficient BackProp

  2012cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• Practical Recommendations for Gradient-Based Training of Deep Architectures

  2012cited by this paper
• Random Search for Hyper-Parameter Optimization

  2012cited by this paper
• No more pesky learning rates

  2012cited by this paper
• Reading Digits in Natural Images with Unsupervised Feature Learning

  2011cited by this paper
• Algorithms for Hyper-Parameter Optimization

  2011cited by this paper
• Understanding the difficulty of training deep feedforward neural networks

  2010cited by this paper
• Learning Multiple Layers of Features from Tiny Images

  2009influential reference
• On the power of small-depth threshold circuits

  1990cited by this paper

• ISRToken: Learning similarities Tokens for precise infrared spectrum recognition model via Transformer

  2023cites this paper
• Intelligent metaheuristic cluster-based wearable devices for healthcare monitoring in telemedicine systems

  2022cites this paper
• EfficientWord-Net: An Open Source Hotword Detection Engine based on One-shot Learning

  2021cites this paper
• Establishment of super sonic inlet flow pattern monitoring system: A workflow

  2021cites this paper
• Pig Weight and Body Size Estimation Using a Multiple Output Regression Convolutional Neural Network: A Fast and Fully Automatic Method

  2021cites this paper
• Real-Time Remote Health Monitoring System Driven by 5G MEC-IoT

  2020cites this paper
• A Review on Recent Progress in Thermal Imaging and Deep Learning Approaches for Breast Cancer Detection

  2020cites this paper
• Continuous-in-Depth Neural Networks

  2020cites this paper
• α-UNet++: A Data-Driven Neural Network Architecture for Medical Image Segmentation

  2020cites this paper
• Pixel-Based Attack on ODENet Classifiers

  2020cites this paper
• Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation

  2019cites this paper
• Deep Model Compression via Two-Stage Deep Reinforcement Learning

  2019cites this paper
• Comparison of different approaches to reduce the number of parameters in Deep Neural Networks

  2019influential citation
• A CNN-based methodology for breast cancer diagnosis using thermal images

  2019influential citation
• Towards Understanding the Importance of Shortcut Connections in Residual Networks

  2019cites this paper
• Dynamic Residual Dense Network for Image Denoising

  2019cites this paper