New massively parallel scheme for Incompressible Smoothed Particle Hydrodynamics (ISPH) for highly nonlinear and distorted flow

A new massively parallel scheme is developed to simulate free-surface flows with the meshless method incompressible smoothed particle hydrodynamics (ISPH) for simulations involving more than 100 million particles. As a pressure-projection method, ISPH requires the solution of a sparse matrix for the pressure Poisson equation (PPE) which is non trivial for large problems where the particles are moving with continuously evolving sparsity. The new scheme uses a Hilbert space filling curve with a cell-linked list to map the entire domain so that domain decomposition and load balancing can be achieved easily to take advantage of geometric locality in order to reduce latency in memory cache access. The computational domain can be subdivided into more than 12,000 partitions using the message passing interface (MPI) for communication between partitions. Load balancing is achieved using the open-source Zoltan library using a new particle weighting system. To solve the PPE for large problems using tens of thousands of partitions, the open-source PETSc library is used which requires the HYPRE BoomerAMG preconditioner to ensure rapid convergence for ISPH. The performance of the code is benchmarked on the U.K. National Supercomputer ARCHER. The results show that domain decomposition with a space filling curve can efficiently treat irregularly distributed particles creating a well-balanced scheme demonstrating that the approach is well matched to the highly irregular subdomains and non-uniform distribution of ISPH free-surface simulations. The benchmark results show that massively parallel ISPH code can achieve over 90% efficiency for the solution of the PPE, but the efficiency of computing matrix coefficients decreases when using more than 12000 partitions giving overall efficiencies in excess of 43% up to 6144 MPI partitions, highlighting future improvements required. The work demonstrates that the Zoltan and PETSc libraries can be effectively combined with ISPH to offer the capability of developing a massively parallel ISPH toolkit.