Learning@home: Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts

No abstract is available for this paper.

• Publication year

  2020

• Venue

  arXiv.org

• Publication date

  2020-02-10

• Fields of study

  Computer Science

• Identifiers

  No identifiers available.

• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Scaling Laws for Neural Language Models

  2020cited by this paper
• 5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

  2020cited by this paper
• Learning from Home

  2020cited by this paper
• PipeMare: Asynchronous Pipeline Parallel DNN Training

  2019cited by this paper
• ZeRO: Memory Optimization Towards Training A Trillion Parameter Models

  2019cited by this paper
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  2019cited by this paper
• Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks

  2019cited by this paper
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

  2019cited by this paper
• PipeDream: generalized pipeline parallelism for DNN training

  2019cited by this paper
• The HSIC Bottleneck: Deep Learning without Back-Propagation

  2019cited by this paper
• RoBERTa: A Robustly Optimized BERT Pretraining Approach

  2019cited by this paper
• Large Memory Layers with Product Keys

  2019cited by this paper
• Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning

  2019cited by this paper
• Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes

  2019cited by this paper
• Natural Compression for Distributed Deep Learning

  2019cited by this paper
• Defending Against Neural Fake News

  2019cited by this paper
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

  2019cited by this paper
• Large Scale Learning of General Visual Representations for Transfer

  2019cited by this paper
• Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD

  2018cited by this paper
• How To Backdoor Federated Learning

  2018cited by this paper
• A hybrid GPU cluster and volunteer computing platform for scalable deep learning

  2018cited by this paper
• Analyzing Federated Learning through an Adversarial Lens

  2018cited by this paper
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

  2018influential reference
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

  2017cited by this paper
• Large-Scale Evolution of Image Classifiers

  2017cited by this paper
• A Case Study of IPv6 Network Performance: Packet Delay, Loss, and Reordering

  2017cited by this paper
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

  2017cited by this paper
• Practical Secure Aggregation for Privacy-Preserving Machine Learning

  2017cited by this paper
• Gaussian Error Linear Units (GELUs)

  2016cited by this paper
• Expert Gate: Lifelong Learning with a Network of Experts

  2016cited by this paper
• Communication-Efficient Learning of Deep Networks from Decentralized Data

  2016cited by this paper
• Decoupled Neural Interfaces using Synthetic Gradients

  2016cited by this paper
• Training Deep Nets with Sublinear Memory Cost

  2016cited by this paper
• Staleness-Aware Async-SGD for Distributed Deep Learning

  2015cited by this paper
• 8-Bit Approximations for Parallelism in Deep Learning

  2015cited by this paper
• ATLAS@Home: Harnessing Volunteer Computing for HEP

  2015cited by this paper
• Deep Residual Learning for Image Recognition

  2015influential reference
• Sybil Nodes as a Mitigation Strategy Against Sybil Attack

  2014cited by this paper
• Dropout: a simple way to prevent neural networks from overfitting

  2014cited by this paper
• Learning Factored Representations in a Deep Mixture of Experts

  2013cited by this paper
• Folding research recruits unconventional help.

  2012cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012influential reference
• Real-world sybil attacks in BitTorrent mainline DHT

  2012cited by this paper
• A survey of DHT security techniques

  2011cited by this paper
• Generating a function for network delay

  2010cited by this paper
• Hierarchical Mixture of Classification Experts Uncovers Interactions between Brain Regions

  2009cited by this paper
• Folding@Home and Genome@Home: Using distributed computing to tackle previously intractable problem

  2009cited by this paper
• ImageNet: A large-scale hierarchical image database

  2009influential reference
• A denial-of-service resistant DHT

  2007cited by this paper
• Nonlinear Models Using Dirichlet Process Mixtures

  2007cited by this paper
• Tapestry: a resilient global-scale overlay for service deployment

  2004cited by this paper
• BOINC: a system for public-resource computing and storage

  2004cited by this paper
• Koorde: A Simple Degree-Optimal Distributed Hash Table

  2003cited by this paper
• Looking up data in P2P systems

  2003cited by this paper
• Kademlia: A Peer-to-Peer Information System Based on the XOR Metric

  2002cited by this paper
• Infinite Mixtures of Gaussian Process Experts

  2001cited by this paper
• GradientBased Learning Applied to Document Recognition

  2001cited by this paper
• Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems

  2001cited by this paper
• A scalable content-addressable network

  2001cited by this paper
• A Parallel Mixture of SVMs for Very Large Scale Problems

  2001cited by this paper
• Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation

  2000cited by this paper
• Beyond Hierarchies : Design Considerations for Distribute d Caching on the Internet

  1998cited by this paper
• Gradient-based learning applied to document recognition

  1998cited by this paper
• Hierarchical Mixtures of Experts and the EM Algorithm

  1993cited by this paper
• Adaptive Mixtures of Local Experts

  1991cited by this paper
• A bridging model for parallel computation

  1990cited by this paper

• Learning to Group Auxiliary Datasets for Molecule

  2023cites this paper
• Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference

  2021cites this paper
• BitTensor: An Intermodel Intelligence Measure

  2020cites this paper
• Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts

  2020cites this paper
• BitTensor: A Peer-to-Peer Intelligence Market

  2020cites this paper