Fixed-point performance analysis of recurrent neural networks

Recurrent neural networks have shown excellent performance in many applications; however they require increased complexity in hardware or software based implementations. The hardware complexity can be much lowered by minimizing the word-length of weights and signals. This work analyzes the fixed-point performance of recurrent neural networks using a retrain based quantization method. The quantization sensitivity of each layer in RNNs is studied, and the overall fixed-point optimization results minimizing the capacity of weights while not sacrificing the performance are presented. A language model and a phoneme recognition examples are used.

• Publication year

  2015

• Venue

  IEEE International Conference on Acoustics, Speech, and Signal Processing

• Publication date

  2015-12-04

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1109/MSP.2015.2411564arXiv 1512.01322
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Single stream parallelization of generalized LSTM-like RNNs on a GPU

  2015cited by this paper
• Fixed point optimization of deep convolutional neural networks for object recognition

  2015cited by this paper
• Show and tell: A neural image caption generator

  2014influential reference
• X1000 real-time phoneme recognition VLSI using feed-forward deep neural networks

  2014cited by this paper
• Fixed-point feedforward deep neural network design using weights +1, 0, and −1

  2014cited by this paper
• Speech recognition with deep recurrent neural networks

  2013cited by this paper
• ADADELTA: An Adaptive Learning Rate Method

  2012cited by this paper
• Generating Text with Recurrent Neural Networks

  2011cited by this paper
• Sequential Deep Learning for Human Action Recognition

  2011cited by this paper
• Extensions of recurrent neural network language model

  2011influential reference
• Hardware accelerated convolutional neural networks for synthetic vision systems

  2010cited by this paper
• The''echo state''approach to analysing and training recurrent neural networks

  2001influential reference
• Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping

  2000cited by this paper
• Neural Network Adaptations to Hardware Implementations

  1997cited by this paper
• Long Short-Term Memory

  1997cited by this paper
• Simulation-based word-length optimization method for fixed-point digital signal processing systems

  1995cited by this paper
• The effects of quantization on multilayer neural networks

  1995cited by this paper
• Finite Precision Error Analysis of Neural Network Hardware Implementations

  1993cited by this paper
• Multilayer feedforward neural networks with single powers-of-two weights

  1993cited by this paper
• DARPA TIMIT:: acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1-1.1

  1993cited by this paper
• A method for self-determination of adaptive learning rates in back propagation

  1991cited by this paper
• Backpropagation Through Time: What It Does and How to Do It

  1990cited by this paper
• Finding Structure in Time

  1990cited by this paper
• Increased rates of convergence through learning rate adaptation

  1987cited by this paper
• A method for unconstrained convex minimization problem with the rate of convergence o(1/k^2)

  1983cited by this paper

• Power Efficient Joint Channel Coding and Frequency Modulation With Deep Learning

  2026cites this paper
• Practical Challenges in Applying QAT to RNN Models

  2025cites this paper
• A Computation-Quantized Training Framework to Generate Accuracy Lossless QNNs for One-Shot Deployment in Embedded Systems

  2025cites this paper
• Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications

  2025cites this paper
• PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models

  2023cites this paper
• A General-Purpose Compiler Design for Instruction-Based AI Accelerator Implementation

  2023cites this paper
• Patch-wise Mixed-Precision Quantization of Vision Transformer

  2023cites this paper
• Exploiting Non-idealities of Resistive Switching Memories for Efficient Machine Learning

  2022cites this paper
• A Hybrid Asymmetric Integer-Only Quantization Method of Neural Networks for Efficient Inference

  2022cites this paper
• Latency-constrained DNN architecture learning for edge systems using zerorized batch normalization

  2022cites this paper
• Squashed Weight Distribution for Low Bit Quantization of Deep Models

  2022cites this paper
• A Post-training Quantization Method for the Design of Fixed-Point-Based FPGA/ASIC Hardware Accelerators for LSTM/GRU Algorithms

  2022cites this paper
• SPIQ: Data-Free Per-Channel Static Input Quantization

  2022cites this paper
• ANNETTE: Accurate Neural Network Execution Time Estimation With Stacked Models

  2021cites this paper
• Characterizing and Accelerating End-to-End EdgeAI Inference Systems for Object Detection Applications

  2021cites this paper
• Filter Pruning via Measuring Feature Map Information

  2021cites this paper
• IVS-Caffe—Hardware-Oriented Neural Network Model Development

  2021cites this paper
• Low Cost Smartband to Prevent COVID-19

  2021cites this paper
• Efficient Spatio-Temporal Modeling Methods for Real-Time Violence Recognition

  2021cites this paper
• Q-Rater: Non-Convex Optimization for Post-Training Uniform Quantization

  2021influential citation
• A Survey of Quantization Methods for Efficient Neural Network Inference

  2021cites this paper
• Reduced-Order Neural Network Synthesis With Robustness Guarantees

  2021cites this paper
• Chapter Eight - Energy-efficient deep learning inference on edge devices

  2021cites this paper
• Unlocking Wordline-level Parallelism for Fast Inference on RRAM-based DNN Accelerator

  2020cites this paper
• Balancing Computation Loads and Optimizing Input Vector Loading in LSTM Accelerators

  2020cites this paper
• A comprehensive survey on model compression and acceleration

  2020cites this paper
• HLHLp: Quantized Neural Networks Training for Reaching Flat Minima in Loss Surface

  2020cites this paper
• The building blocks of a brain-inspired computer

  2020cites this paper
• Resource-Efficient Speech Mask Estimation for Multi-Channel Speech Enhancement

  2020cites this paper
• Learned Low Precision Graph Neural Networks

  2020cites this paper
• Hawkware: Network Intrusion Detection based on Behavior Analysis with ANNs on an IoT Device

  2020cites this paper
• Robust error bounds for quantised and pruned neural networks

  2020cites this paper
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting

  2019influential citation
• Deep Neural Network Approximation for Custom Hardware

  2019cites this paper
• Exploiting Model-Level Parallelism in Recurrent Neural Network Accelerators

  2019cites this paper
• ShapeShifter: Enabling Fine-Grain Data Width Adaptation in Deep Learning

  2019cites this paper
• Overwrite Quantization: Opportunistic Outlier Handling for Neural Network Accelerators

  2019cites this paper
• OverQ: Opportunistic Outlier Quantization for Neural Network Accelerators

  2019cites this paper
• Laconic Deep Learning Inference Acceleration

  2019cites this paper
• Post-Training 4-bit Quantization on Embedding Tables

  2019cites this paper
• Low Complex & High Accuracy Computation Approximations to Enable On-Device RNN Applications

  2019cites this paper
• Sparse Artificial Neural Networks Using a Novel Smoothed LASSO Penalization

  2019cites this paper
• Local stability analysis and H∞ performance for Lipschitz digital filters with saturation nonlinearity and external interferences

  2018cites this paper
• Embedded Neural Networks for Continuous Patient Posture Classification

  2018cites this paper
• On-Device End-to-end Speech Recognition with Multi-Step Parallel Rnns

  2018cites this paper
• Accelerating Inference In Long Short-Term Memory Neural Networks

  2018cites this paper
• Protection of Superconducting Industrial Machinery Using RNN-Based Anomaly Detection for Implementation in Smart Sensor

  2018cites this paper
• Approximate Computing for Long Short Term Memory (LSTM) Neural Networks

  2018cites this paper
• Accurate and Efficient Fixed Point Inference for Deep Neural Networks

  2018cites this paper
• Deep Learning and Reconfigurable Platforms in the Internet of Things: Challenges and Opportunities in Algorithms and Hardware

  2018cites this paper
• Hardware-Oriented Compression of Long Short-Term Memory for Efficient Inference

  2018cites this paper
• Efficient hardware acceleration for approximate inference of bitwise deep neural networks

  2017cites this paper
• Fixed-point optimization of deep neural networks with adaptive step size retraining

  2017cites this paper
• A fast and power efficient architecture to parallelize LSTM based RNN for cognitive intelligence applications

  2017cites this paper
• Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks

  2017cites this paper
• LogNet: Energy-efficient neural networks using logarithmic computation

  2017cites this paper
• Accelerating Recurrent Neural Networks: A Memory-Efficient Approach

  2017cites this paper
• A reduced complexity MFCC-based deep neural network approach for speech enhancement

  2017cites this paper
• Recurrent Neural Networks With Limited Numerical Precision

  2016cites this paper
• FPGA-Based Low-Power Speech Recognition with Recurrent Neural Networks

  2016cites this paper
• Quantized neural network design under weight capacity constraint

  2016cites this paper
• Convolutional Neural Networks using Logarithmic Data Representation

  2016cites this paper
• Architecture exploration of a programmable neural network processor for embedded systems

  2016cites this paper
• Dynamic hand gesture recognition for wearable devices with low complexity recurrent neural networks

  2016cites this paper
• Resiliency of Deep Neural Networks under Quantization

  2015cites this paper