Skill assessment of spatial maps for oceanographic modeling

Comparing data-derived and model-generated spatial maps of key variables is a critical part of oceanographic model skill assessment. Yet, many comparisons resort to presenting the maps side-by-side and saying that the two maps look similar. We describe a suite of techniques for comparing 2-dimensional spatial maps that allow for fuzziness in the comparison and that are included in the Map Comparison Kit software package. These techniques included Kappa, Fuzzy Kappa, Moving Window, and Aggregated Cells. We also introduce a new measure that we call “slip-and-slide.” The slip-and-slide technique is based on determining what has to be done (shift, rotate, re-scale axes) to the model map to get the model map to agree as close as possible to the data map.We simulated two commonly encountered spatial patterns on a 100 by 100 grid of cells (hotspot and gradient), and used the generated maps in two simulation experiments. The first experiment was designed to confirm that the new slip-and-slide method could recover known differences between the data and model maps (identical twin experiment), while the second experiment included all of the map comparison techniques applied to data and model maps that were generated from different response surfaces. The slip-and-slide method successfully recovered the known distortion parameters under ideal conditions and when the response variable differed in magnitude between the model and data maps. Slip-and-slide started to show some difficulty in recovering known distortion parameters when noise was added to the fit of the response surface used in the parameter searching. The second simulation experiment demonstrated the fuzzy techniques, and how the similarity between data and model maps can be quantified and evaluated using increasing degrees of fuzziness in the various techniques. The second simulation experiment also showed the success of the slip-and-slide method, as the various techniques all showed greater similarity between the data and distorted model maps than between the data and the original model maps.Many of the fuzzy methods were designed for comparing maps with many small features, whereas oceanographic maps typically involve large-scale patterns. We propose that skill assessment of spatial maps be a multi-step process: (1) application of the fuzzy methods to the data and original model maps to initially quantify similarity, (2) application of the slip-and-slide method to align the large-scale features of the data and model maps, and (3) re-application of the fuzzy methods to the data and slip-and-slide distorted model map. Skill is quantified by the degree of improvement in similarity due to the slip-and-slide, and the values of the distortion parameters used to slip and slide. We offer some practical tips for how to implement the various skill assessment methods, and mention other metrics and techniques that may be useful for oceanographic model skill assessment. The techniques described should become part of the oceanographic modeling toolbox.