Performance characterization of irregular I/O at the extreme scale

• We statistically model the I/O behavior of two irregular applications.
• We feed the model into two ADIOS I/O kernels and two HDF5 I/O kernels.
• We measure the performance of the two applications’ I/O under different I/O parameter settings.
• We critically compare and contrast the results for a large set of I/O scenarios.
• We show how I/O size and irregular I/O patterns are relevant factors when tuning performance.

This paper reports our experience with irregular I/O and describes lessons learned when running applications with such I/O on supercomputers at the extreme scale. Specifically, we study how irregularities in I/O patterns (i.e., irregular amount of data written per process at each I/O step) in scientific simulations can cause increasing I/O times and substantial loss in scalability. To this end, we quantify the impact of irregular I/O patterns on the I/O performance of scientific applications at the extreme scale by statistically modeling the irregular I/O behavior of two scientific applications: the Monte Carlo application QMCPack and the adaptive mesh refinement application ENZO. For our testing, we feed our model into I/O kernels of two well-known I/O data models (i.e., ADIOS and HDF) to measure the performance of the two applications’ I/O under different I/O settings. Empirically, we show how the growing data sizes and the irregular I/O patterns in these applications are both relevant factors impacting performance.