Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units

Rectifier neuron units (ReLUs) have been widely used in deep convolutional networks. An ReLU converts negative values to zeros, and does not change positive values, which leads to a high sparsity of neurons. In this work, we first examine the sparsity of the outputs of ReLUs in some popular deep convolutional architectures. And then we use the sparsity property of ReLUs to accelerate the calculation of convolution by skipping calculations of zero-valued neurons. The proposed sparse convolution algorithm achieves some speedup improvements on CPUs compared to the traditional matrix-matrix multiplication algorithm for convolution when the sparsity is not less than 0.9.

• Publication year

  2017

• Venue

  arXiv.org

• Publication date

  2017-04-25

• Fields of study

  Computer Science

• Identifiers
  arXiv 1704.07724
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Efficient Processing of Deep Neural Networks: A Tutorial and Survey

  2017cited by this paper
• Convolutional Neural Networks

  2017cited by this paper
• Dynamic Network Surgery for Efficient DNNs

  2016cited by this paper
• Deep Learning

  2016cited by this paper
• Learning Structured Sparsity in Deep Neural Networks

  2016cited by this paper
• Benchmarking State-of-the-Art Deep Learning Software Tools

  2016cited by this paper
• Faster CNNs with Direct Sparse Convolutions and Guided Pruning

  2016cited by this paper
• Minerva: Enabling Low-Power, Highly-Accurate Deep Neural Network Accelerators

  2016cited by this paper
• Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks

  2016cited by this paper
• Learning both Weights and Connections for Efficient Neural Network

  2015cited by this paper
• Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

  2015cited by this paper
• Sparse Convolutional Neural Networks

  2015cited by this paper
• Deep Residual Learning for Image Recognition

  2015cited by this paper
• Fast Algorithms for Convolutional Neural Networks

  2015cited by this paper
• Going deeper with convolutions

  2014influential reference
• DaDianNao: A Machine-Learning Supercomputer

  2014cited by this paper
• cuDNN: Efficient Primitives for Deep Learning

  2014cited by this paper
• Caffe: Convolutional Architecture for Fast Feature Embedding

  2014cited by this paper
• Rectifier Nonlinearities Improve Neural Network Acoustic Models

  2013cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• Deep Sparse Rectifier Neural Networks

  2011cited by this paper
• Rectified Linear Units Improve Restricted Boltzmann Machines

  2010cited by this paper
• Characterizing the sparseness of neural codes

  2001cited by this paper
• Gradient-based learning applied to document recognition

  1998influential reference
• Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition

  1982cited by this paper

• Sparse Activations with Correlated Weights in Cortex-Inspired Neural Networks

  2024cites this paper
• Design of A Sparse Convolutional Neural Network Accelerator

  2024cites this paper
• ReLU, Sparseness, and the Encoding of Optic Flow in Neural Networks

  2024cites this paper
• A Short-Term Traffic Flow Prediction Method for Airport Group Route Waypoints Based on the Spatiotemporal Features of Traffic Flow

  2024cites this paper
• DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models

  2024cites this paper
• FISEdit: Accelerating Text-to-image Editing via Cache-enabled Sparse Diffusion Inference

  2023cites this paper
• Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models

  2022cites this paper
• High Performance Convolution Using Sparsity and Patterns for Inference in Deep Convolutional Neural Networks

  2021cites this paper
• Learning to Sparsify Differences of Synaptic Signal for Efficient Event Processing

  2021cites this paper
• Early Prediction of DNN Activation Using Hierarchical Computations

  2021cites this paper
• NLP-Fast: A Fast, Scalable, and Flexible System to Accelerate Large-Scale Heterogeneous NLP Models

  2021cites this paper
• A Study of Features and Deep Neural Network Architectures and Hyper-Parameters for Domestic Audio Classification

  2021cites this paper
• Dual-side Sparse Tensor Core

  2021cites this paper
• PFA: Privacy-preserving Federated Adaptation for Effective Model Personalization

  2021influential citation
• Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation

  2020cites this paper
• Image Super-Resolution Reconstruction Using Generative Adversarial Networks Based on Wide-Channel Activation

  2020cites this paper
• Sparse Data-Driven Learning for Effective and Efficient Biomedical Image Segmentation.

  2020cites this paper
• GraphChallenge.org Sparse Deep Neural Network Performance

  2020cites this paper
• Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Format

  2020cites this paper
• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights

  2020cites this paper
• Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Driving

  2020cites this paper
• Adaptive Deep Reuse: Accelerating CNN Training on the Fly

  2019cites this paper
• Acorns: A Framework for Accelerating Deep Neural Networks with Input Sparsity

  2019cites this paper
• Sparse Deep Neural Network Graph Challenge

  2019cites this paper
• Performance optimization of convolution calculation by blocking and sparsity on GPU

  2019cites this paper
• Accelerating Convolutional Neural Network by Exploiting Sparsity on GPUs

  2019cites this paper
• Blocking and sparsity for optimization of convolution calculation algorithm on GPUs

  2019cites this paper
• SeerNet: Predicting Convolutional Neural Network Feature-Map Sparsity Through Low-Bit Quantization

  2019cites this paper
• CSCC: Convolution Split Compression Calculation Algorithm for Deep Neural Network

  2019cites this paper
• Sparse Deep Neural Network Exact Solutions

  2018cites this paper
• Recurrent Residual Module for Fast Inference in Videos

  2018influential citation
• SBNet: Sparse Blocks Network for Fast Inference

  2018cites this paper
• 17 Element-Wise Addition and Element-Wise Multipication : Combining Graphs , Intersecting Graphs , Scaling Graphs

  2018cites this paper
• A Novel Business Process Prediction Model Using a Deep Learning Method

  2018cites this paper
• Cascaded Coarse-to-Fine Deep Kernel Networks for Efficient Satellite Image Change Detection

  2018cites this paper
• Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks

  2018cites this paper
• Tissue segmentation of head and neck for hyperthermia treatment planning

  2018cites this paper
• Enabling massive deep neural networks with the GraphBLAS

  2017cites this paper
• Optimization of Adaptive Spiking Neural Networks on GPU

  2017cites this paper
• A study of generative adversarial networks and possible extensions of GANs

  2017cites this paper
• Sparsity Execution time for main loop body vs Inverse sparsity of weight matrix GrB : 512 GrB : 2048 GrB : 8192 GrB : 32768 BLAS

  2017cites this paper
• Bayesian Compression for Deep Learning

  2017cites this paper
• Coarse-to-Fine Deep Kernel Networks

  2017cites this paper
• Spatially Sparse Inference for Generative Image Editing

  year unknowncites this paper