Proactive process-level live migration and back migration in HPC environments

As the number of nodes in high-performance computing environments keeps increasing, faults are becoming common place. Reactive fault tolerance (FT) often does not scale due to massive I/O requirements and relies on manual job resubmission.This work complements reactive with proactive FT at the process level. Through health monitoring, a subset of node failures can be anticipated when one’s health deteriorates. A novel process-level live migration mechanism supports continued execution of applications during much of process migration. This scheme is integrated into an MPI execution environment to transparently sustain health-inflicted node failures, which eradicates the need to restart and requeue MPI jobs. Experiments indicate that 1–6.5 s of prior warning are required to successfully trigger live process migration while similar operating system virtualization mechanisms require 13–24 s. This self-healing approach complements reactive FT by nearly cutting the number of checkpoints in half when 70% of the faults are handled proactively. The work also provides a novel back migration approach to eliminate load imbalance or bottlenecks caused by migrated tasks. Experiments indicate the larger the amount of outstanding execution, the higher the benefit due to back migration.

► Novel process-level live migration is devised and integrated into an MPI runtime system for proactive fault tolerance.
► Live process migration can be triggered as late as 1–6.5 s prior to actual faults under health monitoring.
► This compares favorably to 13–24 s for live migration under operating system virtualization.
► Combined with checkpoints, the number of checkpoints can be cut in half when 70% of faults are handled proactively.
► A novel back migration approach eliminates load imbalance or bottlenecks caused by migrated tasks.