Handling tens of thousands of cores with industrial/legacy codes: Approaches, implementation and timings

The consequences that the recent stagnation in clockrates for CPUs has had on large-scale CFD runs are examined. At first sight, the conclusion is that only massive parallelism at the loop or domain decomposition level will lead to higher FLOP counts. However, the significant differences in advances for CPUs/GPUs versus RAM and interprocessor communication bandwidth lead to a so-called ‘limiting domain size’, below which communication dominates execution times and performance degrades drastically. The consequences of this ‘red-shift’ for the future of CFD are manifold: the time to advance the solution one timestep is limited, implying that even with unlimited number of processors/cores, LES, DES and DNS runs for realistic Reynolds-numbers will require days or weeks of execution.