Software for sharing burst buffers

This project, which started Oct. 2020, is leveraging a suite of technologies, such as disassociation of I/O processing from control logic, time-sliced intra I/O node sharing, function interception for low overhead POSIX I/O, and metadata and data placement for optimal individual application performance. It is investigating how to best apply these technologies, by:

1. Identifying optimal burst buffer configurations for a suite of representative supercomputing applications;
2. Proposing, prototyping, and verifying different design options to address intra-node and inter-node I/O performance sharing;
3. Designing and evaluating a set of sharing policies, such as fair sharing and priority sharing, with real applications and I/O traces.

This project will dramatically increase the sharing capacity of existing burst buffers and enhance domain scientists' productivity at a large scale. It explores various sharing policies that permit efficient sharing of I/O resources and that meet the requirements of computing centers. The results will enable the provisioning of I/O resources, where users can request specific IOPS or bandwidth for a period of time. The prototype burst buffer sharing framework will immediately increase the capacity of existing supercomputers with enhanced I/O performance. The lessons learned will guide next-generation I/O system design for large scale systems. The general improvement of HPC, Big Data, and DL applications will also increase the coherence of the hardware and software used for data analytics computing and modeling and simulation.