Large-scale parallelization based on CPU and GPU cluster for cosmological fluid simulations

• 19–31× speedups on GPU by hierarchical memory optimizations for 5th order WENO scheme.
• Pure CPU parallelization of cosmological fluid simulations that scales well up to 16,384 cores.
• Multi-GPU (up to 256 GPUs) parallelization.
• A detailed analysis of our GPU code on different devices and multi-GPU parallelization.

We present our parallel implementation for large-scale cosmological simulations of 3D supersonic fluids based on CPU and GPU clusters. Our developments are based on a CPU code named WIGEON. It is shown that, compared to the original sequential Fortran code, a speedup of 19–31 (depending on the specific GPU card) can be achieved on single GPU. Furthermore, our results show that the pure MPI parallelization scales very well up to 10 thousand CPU cores. In addition, a hybrid CPU/GPU parallelization scheme is introduced and a detailed analysis of the speedup and the scaling on the different number of CPU/GPU units are presented (up to 256 GPU cards due to computing resource limitation). Our high scalability and speedup rely on the domain decomposition approach, optimization of the algorithm and a series of techniques to optimize the CUDA implementation, especially in the memory access pattern on GPU. We believe this hybrid MPI + CUDA code can be an excellent candidate for 10 Peta-scale computing and beyond.