Session: WORKS20: 15th Workshop on Workflows in Support of Large-Scale Science

Characterizing Scientific Workflows on HPC Systems Using Logs

Wednesday, November 11, 2020

4:30 PM – 5:00 PM ET

Channel: Track 9

Author/Presenter(s)

DG

Devarshi Ghoshal

Lawrence Berkeley National Laboratory, United States of America
BA

Brian Austin

Lawrence Berkeley National Laboratory
DB

Deborah Bard

Lawrence Berkeley National Laboratory
Berkeley, United States of America
CD

Christopher Daley

Lawrence Berkeley National Laboratory, United States of America
Glenn Lockwood

Lawrence Berkeley National Laboratory
Berkeley, United States of America
NW

Nicholas J. Wright

Lawrence Berkeley National Laboratory
berkeley, United States of America
LR

Lavanya Ramakrishnan

Lawrence Berkeley National Laboratory, United States of America

Scientific advances depend on the ability to effectively and efficiently use high performance computing (HPC) systems to manage and run large, complex scientific workflows. Toward understanding the characteristics of these large scientific workflows, we propose two methods to identify workflows with temporal connections and data-dependencies from batch queue and I/O logs available at HPC systems. We use the two methods to characterize and correlate workflow runtime with node requests, I/O patterns, and resource usage on three months of log data available for Cori, a supercomputer at NERSC. A key result from our analyses shows that single-job workflows often do not use all allocated CPUs that provides opportunities to consider allocating resources at a finer-granularity.