Optimization of Collective Communication Operations in MPICH

No abstract is available for this paper.

• Publication year

  2005

• Venue

  The international journal of high performance computing applications

• Publication date

  2005-02-01

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1177/1094342005051521
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• On optimizing collective communication

  2004cited by this paper
• Automatic MPI Counter Profiling of All Users: First Results on a CRAY T3E 900-512

  2004cited by this paper
• A Comparison of MPICH Allgather Algorithms on Switched Networks

  2003cited by this paper
• Communication and Optimization Aspects of Parallel Programming Models on Hybrid Architectures

  2003cited by this paper
• A framework for collective personalized communication

  2003cited by this paper
• Fast collective operations using shared and remote memory access protocols on clusters

  2003cited by this paper
• Recent Advances in Parallel Virtual Machine (PVM) and Message Passing Interface (MPI) - 10th European PVM/MPI Users' Group Meeting, Venice, Italy, September 29 - October 2, 2003, Proceedings

  2003cited by this paper
• The Hierarchical Factor Algorithm for All-to-All Communication (Research Note)

  2002cited by this paper
• On Benchmarking Collective MPI Operations

  2002cited by this paper
• Parallel Computing for the Simulation of 3D Free Surface Flows in Environmental Applications

  2002cited by this paper
• Improved MPI All-to-all Communication on a Giganet SMP Cluster

  2002cited by this paper
• CollMark: MPI Collective Communication Benchmark

  2000influential reference
• Automatically Tuned Collective Communications

  2000cited by this paper
• Exploiting hierarchy in parallel computer networks to optimize collective operation performance

  2000cited by this paper
• A Critical Assessment of LogP : Towards a Realistic Model of Parallel Computation

  2000cited by this paper
• MagPIe: MPI's collective communication operations for clustered wide area systems

  1999cited by this paper
• Optimization of MPI Collectives on Clusters of Large-Scale SMP’s

  1999cited by this paper
• LogGP: Incorporating Long Messages into the LogP Model for Parallel Computation

  1997cited by this paper
• Recent Advances in Parallel Virtual Machine and Message Passing Interface, 14th European PVM/MPI User's Group Meeting, Paris, France, September 30 - October 3, 2007, Proceedings

  1997cited by this paper
• Efficient Algorithms for All-to-All Communications in Multiport Message-Passing Systems

  1997cited by this paper
• Efficient Algorithms for the Reduce-Scatter Operation in LogGP

  1997cited by this paper
• Users guide for mpich, a portable implementation of MPI

  1996cited by this paper
• Fast Collective Communication Libraries, Please

  1995influential reference
• The Communication Challenge for MPP: Intel Paragon and Meiko CS-2

  1994cited by this paper
• Interprocessor collective communication library (InterCom)

  1994influential reference
• Algorithmic trends in computational fluid dynamics; The Institute for Computer Applications in Science and Engineering (ICASE)/LaRC Workshop, NASA Langley Research Center, Hampton, VA, US, Sep. 15-17, 1991

  1993cited by this paper
• LogP: towards a realistic model of parallel computation

  1993cited by this paper
• Global combine on mesh architectures with wormhole routing

  1993cited by this paper
• Complete exchange on a circuit switched mesh

  1992cited by this paper
• Efficient All-to-All Communication Patterns in Hypercube and Mesh Topologies

  1991cited by this paper
• Complete exchange on the iPSC-860

  1991cited by this paper
• Two algorithms for barrier synchronization

  1988cited by this paper
• Image Algebra Techniques for Parallel Image Processing

  1987cited by this paper
• Parallel processing

  1986cited by this paper

• Decentralized Federated Learning in LEO-Satellite-Based IoE Communications: Latency Optimization Under Reliability Constraints

  2026cites this paper
• On the Computation Rate of All-Reduce

  2026cites this paper
• DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce

  2026cites this paper
• High-Performance RoCE-Capable Multicast for Commodity RDMA Datacenters

  2026cites this paper
• Training LLMs with Fault Tolerant HSDP on 100,000 GPUs

  2026cites this paper
• Multi-physics Preconditioning for Thermally Activated Batteries

  2026cites this paper
• CoCoTree: A Computation-Capable Architecture for Collective Communication in Scalable PIM

  2026cites this paper
• Bit-reproducible parallel phylogenetic tree inference

  2026influential citation
• Communication-Avoiding Linear Algebraic Kernel K-Means on GPUs

  2026cites this paper
• Distributed Large Models Training Optimization With Real-Time Wireless Channel Feedback

  2026cites this paper
• ACOS: Arrays of Cheap Optical Switches

  2026cites this paper
• Harvest: Adaptive Photonic Switching Schedules for Collective Communication in Scale-up Domains

  2026cites this paper
• To Reconfigure or Not to Reconfigure: Optimizing All-to-All Collectives in Circuit-Switched Photonic Interconnects

  2026cites this paper
• TidalMesh: Topology-Driven AllReduce Collective Communication for Mesh Topology

  2025cites this paper
• Attention-Level Speculation

  2025cites this paper
• What Every Computer Scientist Needs To Know About Parallelization

  2025influential citation
• Topology-Aware Interleaved All-Reduce Communication for Dragonfly Network

  2025cites this paper
• Joint-Communication Optimal Matrix Multiplication with Asymmetric Memories

  2025cites this paper
• TCC: Token-Based Congestion Control at Transport Layer for RDMA in AI Datacenter Networks

  2025cites this paper
• FMCC-RT: a scalable and fine-grained all-reduce algorithm for large-scale SMP clusters

  2025cites this paper
• Task placement and traffic interleaving for cross-datacenter LLM training over optical networks

  2025cites this paper
• Decentralized Federated Learning with Non-IID Data: Challenges, Trends, and Future Opportunities

  2025cites this paper
• LUMION: Fast Fault Recovery for ML Jobs Using Programmable Optical Fabrics

  2025cites this paper
• MSCCL++: Rethinking GPU Communication Abstractions for Cutting-edge AI Applications

  2025cites this paper
• COSMIC: Enabling Full-Stack Co-Design and Optimization of Distributed Machine Learning Systems

  2025cites this paper
• Toward a Standardized Representation for Deep Learning Collective Algorithms

  2025cites this paper
• System-performance and cost modeling of Large Language Model training and inference

  2025cites this paper
• Rail Optimized PCIe Topologies for LLMs

  2025cites this paper
• Demystifying NCCL: An In-Depth Analysis of GPU Communication Protocols and Algorithms

  2025cites this paper
• DirectReduce: A Scalable Ring AllReduce Offloading Architecture for Torus Topologies

  2025cites this paper
• nncase: An End-to-End Compiler for Efficient LLM Deployment on Heterogeneous Storage Architectures

  2025cites this paper
• A new switch buffer architecture for dragonfly networks

  2025cites this paper
• Collective Communication for 100k+ GPUs

  2025cites this paper
• Poster: On Harnessing Idle Compute at the Edge for Foundation Model Training

  2025cites this paper
• Analyzing Communication Predictability in LLM Training

  2025cites this paper
• When Light Bends to the Collective Will: A Theory and Vision for Adaptive Photonic Scale-up Domains

  2025cites this paper
• Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM

  2025cites this paper
• Short-circuiting Rings for Low-Latency AllReduce

  2025cites this paper
• Pipeline parallelism with reduced network communication for efficient compute-intensive neural network training

  2025cites this paper
• A Network Coding-Based Approach to Floating-Point Sum Reduction

  2025cites this paper
• RingX: Scalable Parallel Attention for Long-Context Learning on HPC

  2025influential citation
• PAARD: Proximity-aware all-reduce communication for dragonfly networks

  2025cites this paper
• Fine-grained Communication Phase based Analytical Performance Modeling and Analysis

  2025cites this paper
• WATOS: Efficient LLM Training Strategies and Architecture Co-Exploration for Wafer-Scale Chip

  2025cites this paper
• Design and Optimization of GPU-Aware MPI Allreduce Using Direct Sendrecv Communication

  2025cites this paper
• Parameterized Algorithms and Parameter Selection for Fast GPU-GPU Collective Communication

  2025cites this paper
• Visualizing MPI Collective Communication

  2025cites this paper
• PIMnet: A Domain-Specific Network for Efficient Collective Communication in Scalable PIM

  2025cites this paper
• Distributed-memory Algorithms for Sparse Matrix Permutation, Extraction, and Assignment

  2025cites this paper
• PCCL: Photonic circuit-switched collective communication for distributed ML

  2025influential citation
• Parameterized Algorithms for Non-uniform All-to-all

  2025cites this paper
• Symphony: Collective Coordination in Multi-Tenant GPU Clusters

  2025cites this paper
• HiPIPE: Adaptive-Chunk Pipeline Scheduling for Hierarchical Collective Communication

  2025cites this paper
• Decentralized Learning in Space: A Framework for Efficient Model Training in LEO Constellations

  2025cites this paper
• T3P: Topology-Tailored Tensor Parallelism

  2025cites this paper
• CROP: Efficient and Robust Multi-Job Placement in Deep Learning Clusters

  2025cites this paper
• A compilation-based approach to performant reduction and redistribution collective communication algorithms

  2025cites this paper
• Efficient AllReduce with Stragglers

  2025influential citation
• Chimera: Communication Fusion for Hybrid Parallelism in Large Language Models

  2025cites this paper
• Minimizing Communication for Parallel Symmetric Tensor Times Same Vector Computation

  2025cites this paper
• ghZCCL: Advancing GPU-aware Collective Communications with Homomorphic Compression

  2025cites this paper
• FRED: A Wafer-scale Fabric for 3D Parallel DNN Training

  2025influential citation
• Pixel-Resolved Long-Context Learning for Turbulence at Exascale: Resolving Small-scale Eddies Toward the Viscous Limit

  2025cites this paper
• Unified Designs of Multi-Rail-Aware MPI Allreduce and Alltoall Operations Across Diverse GPU and Interconnect Systems

  2025cites this paper
• From ATOP to ZCube: Automated Topology Optimization Pipeline and A Highly Cost-Effective Network Topology for Large Model Training

  2025cites this paper
• Optimizing Allreduce Operations for Modern Heterogeneous Architectures with Multiple Processes per GPU

  2025cites this paper
• A parallel all-pairs shortest paths algorithm for dynamic graphs

  2025cites this paper
• Reconfigurability within Collective Communication Algorithms

  2025cites this paper
• veScale: Consistent and Efficient Tensor Programming with Eager-Mode SPMD

  2025cites this paper
• Aurora: Architecting Argonne's First Exascale Supercomputer for Accelerated Scientific Discovery

  2025cites this paper
• Vedrfolnir: RDMA Network Performance Anomalies Diagnosis in Collective Communications

  2025cites this paper
• Topology-Aware All-Reduce Algorithm Synthesizer Based on Deep Reinforcement Learning

  2025cites this paper
• Load Balancing for LLM Traffic via Flow Block

  2025cites this paper
• OmniFed: A Modular Framework for Configurable Federated Learning from Edge to HPC

  2025cites this paper
• SimHost: A Lightweight End-to-End Simulation Framework for HPC Network Systems

  2025cites this paper
• Partial Parameter Updates for Efficient Distributed Training

  2025cites this paper
• Workflow for high-quality visualisation of large-scale CFD simulations by volume rendering

  2025cites this paper
• TASP: Topology-aware Sequence Parallelism

  2025cites this paper
• Communication-Efficient, 2D Parallel Stochastic Gradient Descent for Distributed-Memory Optimization

  2025cites this paper
• FHE4DMM: A Low-Latency Distributed Matrix Multiplication With Fully Homomorphic Encryption

  2025cites this paper
• Deadlock-Free Routing for Full-Mesh Networks Without Using Virtual Channels

  2025cites this paper
• Optimizing All-to-All Collective Communication with Fault Tolerance on Torus Networks

  2025influential citation
• Dissecting the NVIDIA Hopper Architecture through Microbenchmarking and Multiple Level Analysis

  2025cites this paper
• OMB-Compr: An Extension to OSU Micro Benchmarks for Collective Compression Error Measurement

  2025cites this paper
• LLMEmu: A lightweight performance emulator for high-fidelity distributed LLM training

  2025cites this paper
• Keeping up with Large Language Models: A Holistic Methodology of Compute, Memory, Communication, and Cost Modeling

  2025cites this paper
• State of the Art in Parallel and Distributed Systems: Emerging Trends and Challenges

  2025cites this paper
• Global-QSGD: Allreduce-Compatible Quantization for Distributed Learning with Theoretical Guarantees

  2025cites this paper
• Reunion: Receiver-driven network load balancing mechanism in AI training clusters

  2025cites this paper
• VSS Challenge Problem: Verifying the Correctness of AllReduce Algorithms in the MPICH Implementation of MPI

  2025cites this paper
• Performance Analysis and Predictive Modeling of MPI Collective Algorithms in Multi-Core Clusters: A Comparative Study

  2025cites this paper
• OmniLearn: A Framework for Distributed Deep Learning Over Heterogeneous Clusters

  2025cites this paper
• Routing for Large ML Models

  2025cites this paper
• ZCCL: Significantly Improving Collective Communication With Error-Bounded Lossy Compression

  2025cites this paper
• RAM-Wafer: RL-Based Automatic Mapping Framework for Large-Scale AI Training on Wafer-Scale Computing

  2025cites this paper
• Scalable Dual Coordinate Descent for Kernel Methods

  2024cites this paper
• FRED: Flexible REduction-Distribution Interconnect and Communication Implementation for Wafer-Scale Distributed Training of DNN Models

  2024cites this paper
• Collective Communication Performance Evaluation for Distributed Deep Learning Training

  2024cites this paper
• ALEPH: Accelerating Distributed Training With eBPF-Based Hierarchical Gradient Aggregation

  2024cites this paper
• SmartFuse: Reconfigurable Smart Switches to Accelerate Fused Collectives in HPC Applications

  2024cites this paper