Hybrid MPI-thread parallelization of adaptive mesh operations

• Development of a hybrid MPI-thread programming system called PCU.
• Inter-thread message passing, including non-blocking collectives.
• A novel, scalable termination detection technique for communication rounds.
• Hybrid parallel scalability to 16K cores on an IBM Blue Gene/Q.

Many of the world’s leading supercomputer architectures are a hybrid of shared memory and network-distributed memory. Such an architecture lends itself to a hybrid MPI-thread programming model. We first present an implementation of inter-thread message passing based on the MPI and pthread libraries. In addition, we present an efficient implementation of termination detection for communication rounds. We use the term phased message passing to denote the communication interface based on this termination detection. This interface is then used to implement parallel operations for adaptive unstructured meshes, and the performance of resulting applications is compared to pure MPI operation. We also present new workflows enabled by the ability to vary the number of threads during runtime.