Three-level hybrid vs. flat MPI on the Earth Simulator: Parallel iterative solvers for finite-element method

An efficient parallel iterative method for finite element method has been developed for symmetric multiprocessor (SMP) cluster architectures with vector processors such as the Earth Simulator. The method is based on a three-level hybrid parallel programming model, including message passing for inter-SMP node communication, loop directives by OpenMP for intra-SMP node parallelization and vectorization for each processing element (PE). Simple 3D linear elastic problems with more than 2.2×109 DOF have been solved using 3×3 block ICCG(0) method with additive Schwarz domain decomposition and PDJDS/CM-RCM reordering on 176 nodes of the Earth Simulator, achieving performance of 3.80 TFLOPS. The PDJDS/CM-RCM reordering method provides excellent vector and parallel performance in SMP nodes. A three-level hybrid parallel programming model outperforms flat MPI in the problems involving large numbers of SMP nodes.