People Publications Projects

International Collaboration Framework for Extreme Scale Experiments (ICEE)

Large-scale scientific exploration in high-energy physics, fusion, climate and so on are based on international collaborations. As these collaborations produce more and more data at faster rates, the existing workflow management systems are hard pressed to keep pace. A necessary solution is to process, analyze, summarize and reduce the data before it reaches the relatively slow disk storage system. This approach is generally known as in-transit processing or in-flight analysis. The ICEE project aims to introduce this in-transit analysis capability into a collaborative workflow system by leveraging the in-transit capability of ADIOS and selective data access capability of FastBit. Additionally, ICEE seeks to provide an effective data flow management for distributed workflows and enable large international projects to make near real-time collaborative decisions.