Applying Parallel Computing Techniques to Analyze Terabyte Atmospheric Boundary Layer Model Outputs

In the atmospheric sciences, the size of simulation output continues to grow as computational resources able to handle simulations with fine-scale spatial and temporal resolutions become more accessible. As output size increases, serial data analysis methods become overwhelmed, resulting in either long delays during processing or total failures due to memory constraints. Parallel data analysis methods can alleviate these issues, however atmospheric scientists are often unfamiliar with how to achieve this. Therefore, example methods are needed to help guide the use of parallel processing in the analysis of Big Data from atmospheric simulations.In this work, practical methods are presented by which an analysis may be executed in parallel using the Message Passing Interface (MPI) and Python. These methods first consider the inherent spatial dependencies of a particular data analysis process. By identifying these dependencies, horizontal or vertical distribution of the dataset across processes can be carried out with minimal process intercommunication. In addition, an analysis method is classified as either data-transfer-limited or computationally-limited. In data-transfer-limited problems, data transfer time outweighs processing time. In computationally-limited problems, processing time outweighs data transfer time.The results show that by increasing processor count, the execution time of computationally-limited problems shows improvement. For data-transfer-limited problems, increasing node count offers the greatest improvement. To further improve the performance of computationally-limited problems, a Graphics Processing Unit (GPU) and the Compute Unified Device Architecture (CUDA) framework are used. It is shown that this GPU implementation offers further improvement over the MPI version of the analysis methods tested.