[View PDF]


Prabhat (PI), Suren Byna, Oliver Rubel, John Wu, LBNL


  • Ability to analyze very large datasets quickly to enhance scientific understanding and discovery
  • Enhance I/O on HDF5, a popular file format used by many application domains
  • Demonstrate these capabilities on Trillion particle plasma physics simulation


  • Trillion particle plasma physics simulation conducted on 120,000 cores @NERSC
  • Enhanced Parallel HDF5 obtained peak 35GB/s, and 80% sustained I/O rate
  • FastBit was used to index 30TB timestep in 10 minutes and query in 3 seconds


  • Software enabled scientists to search and gain insights from the trillion particle dataset for the first time:
    • Confinement of energetic particles by the flux ropes
    • Asymmetric distribution of particles near the reconnection hot-spot



Magnetic reconnection from a plasma physics simulation (Left). Scientists were able to query and find an asymmetric distribution of particles near the reconnection event (Right) using our software tools.

Surendra Byna et al, "Parallel I/O, Analysis, and Visualization of a Trillion Particle Simulation". SuperComputing conference, SC’12, November 2012.


The slide highlights recent accomplishments from the ExaHDF5 project with collaboration with SDAV staff.

1) Parallel I/O with HDF5
We ran a Trillion particle simulation on 120K cores on hopper. The code produced 30 TB of particle data *per timestep*- To the best of our knowledge, this is the first time that anyone has demonstrated writes to a single, shared 30 TB HDF5 file

  • We hit peak I/O rates on hopper (~35GB/s) for brief time intervals during the run, we made an average ~23GB/s, which is a new record for parallel HDF5 performance

2) FastBit based analysis-

  • We developed a novel hybrid parallel version of FastBit to do the indexing/querying on the dataset
  • This was the first time that we used FastBit and FastQuery to index and query a dataset with Trillion entries
  • We were able to index the dataset in 10 minutes and query the dataset in 3 seconds

3) Scientific insights

  • This is the first time that our science collaborators have been able to examine the trillion particle dataset. They had largely ignored the particle data, or looked at a coarse grained version earlier
  •  Our collaborators had made a number of conjectures and hypothesis regarding the interplay between particles and the magnetic fields and multi-dimensional phase-space distribution of particles. Using these new tools, they were able to confirm these hypothesis quantitatively. More specifically the scientists found:
    • a preferential acceleration of particles in a direction parallel to the magnetic field
    • energetic particles carrying a significant amount of current, even at early timesteps in the simulation
    • predominant distribution of energetic particles in the current sheet,  suggesting that flux ropes can confine these particles
    • agyrotropic (asymmetric) distribution of particles near the magnetic reconnection event

DOE researchers: Prabhat (PI), Suren Byna, Oliver Rubel and John Wu (LBNL)
Scientific collaborators: Homa Karimabadi (UCSD), Vadim Roytershteyn (UCSD) and Bill Daughton (LANL)
Simulation code used in the study is VPIC, developed at LANL.