Scheduling and flexible control of bandwidth and in-transit services for end-to-end application workflows

• A flow-based in-transit data staging and processing approach.
• An opportunistic QoS model, based on convex optimization, for resource scheduling.
• SDN controller for enabling in-transit services and end-to-end workflow couplings.
• An evaluation using a Fusion workflow scenario and an emulated virtualized SDN testbed.
• The approach achieves high resource utilization and satisfies workflow QoS requirements.

End-to-end scientific application workflows that integrate high-end experiments and instruments with large scale simulations and end-user displays are becoming increasingly important. These workflows require complex couplings and data sharing between distributed components involving large data volumes and present varying hard (in-time data delivery) and soft (in-transit processing) quality of service (QoS) requirements. As a result, supporting efficient data transport is critical for such workflows. In this paper, we leverage software-defined networking (SDN) to address issues of data transport service control and resource provisioning to meet varying QoS requirements from multiple coupled workflows sharing the same service medium. Specifically, we present a flexible control and a disciplined resource scheduling approach for data transport services for science networks. Furthermore, we emulate an SDN testbed on top of the FutureGrid virtualized testbed and use it to evaluate our approach for a realistic scientific workflow. Our results show that SDN-based control and resource scheduling based on simple intuitive models can meet the requirements of the targeted workflows with high resource utilization.