Full-waveform inversion on heterogeneous HPC systems

We present a spectral-element implementation of full seismic waveform inversion for large heterogeneous HPC systems. In this we address the optimal parallelisation configurations of individual simulations, the large I/O requirements of adjoint simulations, and the scheduling of large numbers of forward and adjoint solves, typical for realistic inversions. Using GPU accelerators allows us to achieve a 3.5-4 times performance improvement over the best known homogeneous implementation. We achieve GPU memory throughput varying from 60 to 80% of the peak bandwidth, thus providing good utilisation of hardware resources. We demonstrate the practical applicability of our developments in a real-data application in the western Mediterranean. With the help of GPU accelerators, we are able to model and invert seismic wave propagation in a frequency band that is broader than permitted by the use of CPUs alone, which helps bridging the traditional gap between crustal and mantle tomography.