Next Generation Workload Management and Analysis System for Big Data
The BigPanDA project was established as an extension of the PanDA Project when DOE ASCR and DOE HEP funded in 2012 a three year work program to develop a next generation workload management and analysis system for big data building on PanDA.
PanDA was originally designed specifically for the needs of the ATLAS Experiment at the Large Hadron Collider (LHC), and has proved to be highly successful in meeting all the distributed computing needs of the experiment. The core design of PanDA is however not experiment specific.
PanDA is capable of meeting the needs of other data intensive scientific applications, and the BigPanDA project was created to undertake this. The project has generalized PanDA for use by other experiments, extended its reach to High Performance Computing (HPC) platforms, and added intelligent network awareness to the system.
The BigPanDA work program consists of four work packages as follows.
Factorizing the core components of PanDA to enable adoption by a wide range of exascale scientific communities.
Evolving PanDA to support extreme scale computing clouds and Leadership Computing Facilities.
Leveraging intelligent networks): Integrating network services and real-time data access to the PanDA workflow.
Real time monitoring and visualization package for PanDA.
DOE ASCR and DOE HEP funded the proposal “A next generation workload management and analysis system for big data”, beginning in 2012, with lead PIs Alexei Klimentov (BNL) and Kaushik De (UTA).