Revisiting locality-awareness in view of dynamically changing topologies

As a general rule, when writing parallel applications according to the MPI standard, the programmer does not need to worry about the underlying hardware topology. This is because the MPI standard intentionally hides the actual hardware topology from the application programmer for the seizure of portability, while at the same time burdening the MPI implementation to handle hardware-related peculiarities as optimal as possible. So, for instance, with the emergence of SMP systems, locality-awareness in terms of the recognition of accelerated node-internal communication found its way into all major MPI libraries in the early 2000s. However, the actually implemented degree of such a locality-awareness can vary: From the simple usage of point-to-point communication over shared-memory, via the smart adaptation of collective communication patterns, through to the exploitation of direct accessible address spaces for one-sided communication. Until now, all these locality-related optimizations basically assume a static hierarchical topology in the course of a parallel program. In contrast, this article strives for a discussion of how dynamically changing topologies, as they may result from process migrations, can be considered during runtime for locality-awareness. In doing so, the article focuses on collective communication, but also discusses the challenges for point-to-point and one-sided communication.