ExaHDF5

Delivering Efficient Parallel I/O
on Exascale Computing Systems

Exascale Computing Project (ECP)
Software Technology area

Collaborator Our POC Collab. POC Status / Notes
ECP AD and Co-design projects


Computing the Sky at Extreme Scales Venkat Vishwanath, Suren Byna Salman Habib Understanding current implementation of generic I/O, performance of HDF5 implementation of generic I/O
Lattice Gauge Theory Jialin Liu Chulwoo Jung

Useful features:

  • Data Elevator for write caching
  • Subfiling to create HDF5 subfiles
  • Querying
Subsurface Alex Sim, Suren Byna, Quincey Koziol Brian van Straalen

Chombo I/O currently uses HDF5

  • Track any performance issues
  • Explore usage of features proposed in the ExaHDF5 project
Exascale Modeling of Advanced Particle Accelerators Alex Sim Jean-Luc Vay
  • Performance tuning of lab-frame data write pattern
  • Exploration of Data Elevator-based solution for "boosted frame" data write pattern
  • AMReX I/O to write AMR data
QMCPACK: A Framework for Predictive and Systematically Improvable Quantum-Mechanics Based Simulations of Materials Venkat Vishwanath

NWChemEx Project Venkat Vishwanath, Suren Byna Ray Bair, Sriram Krishnamoorthy Exploring potential collaboration
Data Analyitics at the Exascale for Free Electron Lasers Jialin Liu Amedeo Perazzo
  • SWMR and Virtual Data Sets
Exascale Deep Learning and Simulation Enabled Precision Medicine for Cancer Venkat Vishwanath Venkat Vishwanath Venkat Vishwanath is on both projects and will track the I/O needs of the CANDLE project
1.2.1.15 ADSE15-ACME-MMF ACME-MMF Venkat Vishwanath, Suren Byna Jayesh Krishna
Block Structured AMR Co-design Center Alex Sim, Suren Byna, Quincey Koziol

 Vincent Beckner, Ann Almgren
  • AMReX's I/O
  • Exploring an implementation of I/O interface
  • Data Elevator may be useful
  • Anticipated shared-fate milestone:
    • AMReX will provide the ability for AMReX-based codes to write mesh and particle data in the existing HDF5 format, and will incorporate new ExaHDF5 functionality as appropriate.oExaHDF5 will support AMReX’s efforts in the form of availability for answering questions and in-person consulting.
    • In addition, ExaHDF5 will partner with AMReX to improve the performance of the HDF5 output in the event that it compares unfavorably to the AMReX native output format.
High Performance, Multidisciplinary Simulations for Regional Scale Earthquake Hazard and Risk Assessments Alex Sim, Suren Byna Hans Johansen
  • Exploring HDF5 output
Multiscale Coupled Urban Systems Jialin Liu , Suren Byna Tianzhen Hong
  • Exploring HDF5 output
ECP ST Projects


Data Libraries and Services Enabling Exascale Science Suren Byna Rob Ross
  • Interoperability of HDF5 and netCDF/PnetCDF file formats
  • Use of Darshan for profiling I/O performance
ADIOS Suren Byna Scott Klasky, John Wu
  • Interoperability of HDF5 and ADIOS file formats
UNIFYCR: A Checkpoint/Restart File System for Distributed Burst Buffers Suren Byna Kathryn Mohror
  • Exploring use of node-local file system for write caching
ZFP: Compressed Floating-Point Arrays Suren Byna Peter Lindstrom
  • ZFP compression filter HDF5 data
EZ: Fast, effective, parallel error-bounded exascale lossy compression for scientific data Suren Byna Frank Cappello
  • SZ compression filter HDF5 data