ArrayUDF: Custom Analyses with Automated Data Management

Open source at: https://bitbucket.org/arrayudf/

Challenge

Scientific data analysis code spends a lot of effort on data management and other common tasks, but performs a wide variety of operations. Can we automate the data management without restricting analysis operations?

Solution

Demonstrate a novel scalable framework to perform user-defined custom data analysis on massive datasets on supercomputers

Signficance and Impact

Tens to thousands of times faster than the state-of-the-art Big Data systems

Research Details

• An efficient and scalable user-defined function (UDF) execution framework that allows users to define their operations on the familiar array data
• Optimized data management and execution on supercomputers; scale to larger datasets where Spark ran out of memory
• Tested on operations from large DOE science applications