DynaQuant: Compressing Deep Learning Training Checkpoints via Dynamic Quantization

No abstract is available for this paper.

• Publication year

  2023

• Venue

  arXiv.org

• Publication date

  Unknown publication date

• Fields of study

  Computer Science

• Identifiers
  DOI 10.48550/arXiv.2306.11800
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Design of a Quantization-Based DNN Delta Compression Framework for Model Snapshots and Federated Learning

  2023influential reference
• Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

  2023influential reference
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

  2022cited by this paper
• OPT: Open Pre-trained Transformer Language Models

  2022influential reference
• A Sensitivity-based Pruning Method for Convolutional Neural Networks

  2022cited by this paper
• CheckFreq: Frequent, Fine-Grained DNN Checkpointing

  2021cited by this paper
• Distribution-aware Adaptive Multi-bit Quantization

  2021cited by this paper
• SeReNe: Sensitivity-Based Regularization of Neurons for Structured Sparsity in Neural Networks

  2021cited by this paper
• Hessian-Aware Pruning and Optimal Neural Implant

  2021cited by this paper
• When to Prune? A Policy towards Early Structural Pruning

  2021cited by this paper
• HAWQ-V3: Dyadic Neural Network Quantization

  2021cited by this paper
• Layer Importance Estimation with Imprinting for Neural Network Quantization

  2021cited by this paper
• DeepFreeze: Towards Scalable Asynchronous Checkpointing of Deep Learning Models

  2020cited by this paper
• GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference

  2020influential reference
• Post-training Piecewise Linear Quantization for Deep Neural Networks

  2020cited by this paper
• Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence

  2020influential reference
• Scaling Laws for Neural Language Models

  2020cited by this paper
• ZeroQ: A Novel Zero Shot Quantization Framework

  2020cited by this paper
• A Decade Survey of Transfer Learning (2010–2020)

  2020cited by this paper
• Hessian-Driven Unequal Protection of DNN Parameters for Robust Inference

  2020cited by this paper
• Comparing Fisher Information Regularization with Distillation for DNN Quantization

  2020cited by this paper
• Adaptive Gradient Quantization for Data-Parallel SGD

  2020cited by this paper
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

  2020influential reference
• Degree-Quant: Quantization-Aware Training for Graph Neural Networks

  2020cited by this paper
• On Efficient Constructions of Checkpoints

  2020influential reference
• Delta-DNN: Efficiently Compressing Deep Neural Networks via Exploiting Floats Similarity

  2020influential reference
• Progressive Skeletonization: Trimming more fat from a network at initialization

  2020cited by this paper
• Language Models are Few-Shot Learners

  2020influential reference
• Post-Training 4-bit Quantization on Embedding Tables

  2019cited by this paper
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  2019influential reference
• Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads

  2019cited by this paper
• Low-bit Quantization of Neural Networks for Efficient Inference

  2019cited by this paper
• The State of Sparsity in Deep Neural Networks

  2019cited by this paper
• How much can k-means be improved by using better initialization and repeats?

  2019cited by this paper
• Searching for MobileNetV3

  2019influential reference
• HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision

  2019cited by this paper
• Language Models are Unsupervised Multitask Learners

  2019influential reference
• A Study of BFLOAT16 for Deep Learning Training

  2019cited by this paper
• DDSketch: A Fast and Fully-Mergeable Quantile Sketch with Relative-Error Guarantees

  2019influential reference
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

  2019cited by this paper
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

  2019influential reference
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

  2019cited by this paper
• HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks

  2019cited by this paper
• Adaptive Loss-Aware Quantization for Multi-Bit Networks

  2019cited by this paper
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

  2018cited by this paper
• Learning Compression from Limited Unlabeled Data

  2018cited by this paper
• SNIP: Single-shot Network Pruning based on Connection Sensitivity

  2018cited by this paper
• Optimization based Layer-wise Magnitude-based Pruning for DNN Compression

  2018cited by this paper
• Post training 4-bit quantization of convolutional networks for rapid-deployment

  2018cited by this paper
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

  2018cited by this paper
• HAQ: Hardware-Aware Automated Quantization With Mixed Precision

  2018cited by this paper
• Deep Learning with Low Precision by Half-Wave Gaussian Quantization

  2017cited by this paper
• Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

  2017cited by this paper
• CuPy : A NumPy-Compatible Library for NVIDIA GPU Calculations

  2017cited by this paper
• To prune, or not to prune: exploring the efficacy of pruning for model compression

  2017cited by this paper
• Pruning Convolutional Neural Networks for Resource Efficient Inference

  2016cited by this paper
• Convolutional Neural Networks using Logarithmic Data Representation

  2016cited by this paper
• Towards the Limit of Network Quantization

  2016cited by this paper
• Pointer Sentinel Mixture Models

  2016cited by this paper
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

  2016cited by this paper
• Hardware-oriented Approximation of Convolutional Neural Networks

  2016cited by this paper
• Binarized Neural Networks

  2016cited by this paper
• Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

  2015cited by this paper
• Deep Residual Learning for Image Recognition

  2015influential reference
• Learning both Weights and Connections for Efficient Neural Network

  2015cited by this paper
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations

  2015cited by this paper
• Neural Networks with Few Multiplications

  2015cited by this paper
• Clustering in the Presence of Background Noise

  2014cited by this paper
• Very Deep Convolutional Networks for Large-Scale Image Recognition

  2014cited by this paper
• A Survey on Transfer Learning

  2010cited by this paper
• Clusterability: A Theoretical Study

  2009cited by this paper
• ImageNet: A large-scale hierarchical image database

  2009cited by this paper
• k-means++: the advantages of careful seeding

  2007cited by this paper
• A new pruning heuristic based on variance analysis of sensitivity information

  2001cited by this paper
• Optimal Brain Surgeon and general network pruning

  1993cited by this paper
• Second Order Derivatives for Network Pruning: Optimal Brain Surgeon

  1992cited by this paper
• Optimal Brain Damage

  1989cited by this paper
• Run-length encodings (Corresp.)

  1966cited by this paper
• A method for the construction of minimum-redundancy codes

  1952influential reference

• FlowCheck: Decoupling Checkpointing and Training of Large-Scale Models

  2025cites this paper
• An Efficient Compression of Deep Neural Network Checkpoints Based on Prediction and Context Modeling

  2025cites this paper
• BitSnap: Checkpoint Sparsification and Quantization in LLM Training

  2025influential citation
• TranLogs: Lossless Failure Recovery Empowered by Training Logs

  2024cites this paper
• IncrCP: Decomposing and Orchestrating Incremental Checkpoints for Effective Recommendation Model Training

  2024cites this paper