Intelligent data staging with overlapped execution of grid applications

Existing data grid scheduling systems handle huge data I/O via replica location services coupled with simple staging and decoupled from scheduling of computing tasks. However, when the application/workflow scales, we observe considerable degradations in performance, compared to processing within a tightly-coupled cluster. For example, when numerous nodes access the same set of files simultaneously, extreme performance degradation occurs even if replicas are used, due to bottlenecks that show in the infrastructure. Instead of resorting to expensive solutions such as parallel file systems, we propose tightly coupling replica and data transfer management with computation scheduling for alleviating such situations. In particular, we propose three techniques: (1) data-staging requests aggregation and O(1) replication across multiple nodes using a multireplication framework, (2) replica-centric scheduling, which reuses previously used data for minimizing staging time and (3) overlapped execution of data staging and compute bound tasks. Early benchmark results implemented in our prototype Condor-like grid scheduling system demonstrate that the techniques are quite effective in eliminating much of the overhead in data transfers and achieving 100% of CPU utilization.