Oak Ridge National Laboratory (ORNL), United States of America
As high-performance computing (HPC) is being scaled up to exascale to accommodate new modeling and simulation needs, I/O has continued to be a major bottleneck in end-to-end scientific processes. This work aims to take advantage of the storage characteristics and explore application level solutions that are interference-aware. In particular, we monitor the performance of data analytics and estimate the state of shared storage resources using discrete fourier transform. If there is heavy I/O interference, data analytics can dynamically adapt to the environment by lowering the accuracy and performing partial or no augmentation from the shared storage, dictated by an augmentation-bandwidth plot. We evaluate three data analytics; XGC, GenASiS and Jet; on Chameleon, and quantitatively demonstrate that both the average and variation of I/O performance can be vastly improved, with the mean and variance improved by as much as 18x and 60x, respectively, while maintaining acceptable outcome of data analysis.