Facilitating In-Situ Analytics for Complex AMR-based Simulation Workflows

[View PDF]

 

Manish Parashar, Rutgers

Objective

  • Manage dynamic data processing requirements at extreme scales using coordinated algorithm, middleware and resource layer adaptations

Target Applications

  • Dynamic AMR-based simulations such as the Polytropic Gas simulation for modeling tokomak edge plasma (part of Chombo developed by LBNL)

Data Management Challenges

  • Large and dynamically changing data volumes
  • Dynamic and imbalanced data distributions
  • Heterogeneous and dynamic resource (memory, CPU, etc.) requirements

Impact

  • Accelerated the data-to-insights process by up to 75% for a large-scale AMR-based simulation-analytic workflow
  • Reduced overall data movement between the AMR-based simulation and in-situ analytics by 45%

Publication

T. Jin, F. Zhang, Q. Sun, H. Bui, M. Parashar, H. Yu, S. Klasky, N. Podhorszki, H. Abbasi, “Using Cross-Layer Adaptations for Dynamic Data Management in Large Scale Coupled Scientific Workflows”, SC 2013.

Data automatically translated from full resolution (left) to the reduced resolution (right) to meet the limited memory availability.(Above) Data automatically translated from full resolution (left) to  the reduced resolution (right) to meet the limited memory availability.

(Right) Data movement reduced by 45% for a 3D AMR Advection-Diffusion simulation-analytic workflow using adaptive analytic placement, as compared to in-transit processing.

 Data movement reduced by 45% for a 3D AMR Advection-Diffusion simulation-analytic workflow using adaptive analytic placement, as compared to in-transit processing.