Parallel Data Object Creation: Towards Scalable Metadata Management in High-Performance I/O Library

High-level I/O libraries, such as HDF5 and PnetCDF, are commonly used by large-scale scientific applications to perform I/O tasks in parallel. These I/O libraries store the metadata such as data types and dimensionality along with the raw data in the same files. While these libraries are well-optimized for concurrent access to the raw data, they are designed neither to handle a large number of data objects efficiently nor to create different data objects independently by multiple processes, as they require applications to call data object creation APIs collectively with consistent metadata among all processes. Applications that process data gathered from remote sensors, such as particle collision experiments in high-energy physics, may generate data of different sizes from different sensors and desire to store them as separate data objects. For such applications, the I/O library's requirement on collective data object creation can become very expensive, as the cost of metadata consistency check increases with the metadata volume as well as the number of processes. To address this limitation, using PnetCDF as an experimental platform, we investigate solutions in this paper that abide the netCDF file format, as well as propose a new file header format that enables independent data object creation. The proposed file header consists of two sections, an index table and a list of metadata blocks. The index table contains the reference to the metadata blocks and each block stores metadata of objects that can be created collectively or independently. The new design achieves a scalable performance, cutting data object creation times by up to 582x when running on 4096 MPI processes to create 5,684,800 data objects in parallel. Additionally, the new method reduces the memory footprints, with each process requiring an amount of memory space inversely proportional to the number of processes.

• Publication year

  2025

• Venue

  arXiv.org

• Publication date

  2025-06-18

• Fields of study

  Computer Science, Engineering

• Identifiers
  arXiv 2506.15114
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Hybrid-LLM-GNN: Integrating Large Language Models and Graph Neural Networks for Enhanced Materials Property Prediction

  2024cited by this paper
• A Case Study of Data Management Challenges Presented in Large-Scale Machine Learning Workflows

  2023cited by this paper
• Instant Graph Neural Networks for Dynamic Graphs

  2022cited by this paper
• Atomistic Line Graph Neural Network for improved materials property predictions

  2021cited by this paper
• Accelerating HDF5 I/O for Exascale Using DAOS

  2021cited by this paper
• Evolutionary dynamics of higher-order interactions in social networks

  2020cited by this paper
• Temporal Graph Networks for Deep Learning on Dynamic Graphs

  2020cited by this paper
• Learning Mesh-Based Simulation with Graph Networks

  2020cited by this paper
• Learning Continuous System Dynamics from Irregularly-Sampled Partial Observations

  2020cited by this paper
• Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks

  2019cited by this paper
• Session-Based Social Recommendation via Dynamic Graph Attention Networks

  2019cited by this paper
• Graph WaveNet for Deep Spatial-Temporal Graph Modeling

  2019cited by this paper
• Attention Based Spatial-Temporal Graph Convolutional Networks for Traffic Flow Forecasting

  2019cited by this paper
• A graph-convolutional neural network model for the prediction of chemical reactivity

  2018cited by this paper
• Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting

  2017cited by this paper
• Parallel netCDF: A High-Performance Scientific I/O Interface

  2003cited by this paper

• No citing papers are available for this paper.