Characterizing and profiling scientific workflows

Researchers working on the planning, scheduling, and execution of scientific workflows need access to a wide variety of scientific workflows to evaluate the performance of their implementations. This paper provides a characterization of workflows from six diverse scientific applications, including astronomy, bioinformatics, earthquake science, and gravitational-wave physics. The characterization is based on novel workflow profiling tools that provide detailed information about the various computational tasks that are present in the workflow. This information includes I/O, memory and computational characteristics. Although the workflows are diverse, there is evidence that each workflow has a job type that consumes the most amount of runtime. The study also uncovered inefficiency in a workflow component implementation, where the component was re-reading the same data multiple times.

► The workflows of six diverse scientific applications are characterized.
► The characterization includes workflow structure as well as I/O, memory and CPU usage.
► We describe new techniques that were developed to profile scientific workflows.
► The information provided can be used to create realistic synthetic workflows for use in simulation studies of workflow systems.