Indices for the evaluation of wildfire spread simulations using contemporaneous predictions and observations of burnt area

Methods to objectively evaluate performance are critical for model development. In contrast to recent advances in wildfire simulation, there has been limited attention to evaluating fire model performance. Information to validate fire models is typically limited, commonly to a few perimeter observations at a small number of points in time. We review metrics for comparing two burnt areas at a point in time: observed and predicted. These are compared in an idealised landscape and with a case study evaluating the performance of simulations of an Australian wildfire. We assessed: Shape Deviation Index (SDI), Jaccard's coefficient, F1, Sørensen's Similarity and Area Difference Index (ADI). For decomposing fit into error components (overprediction and underprediction) we assessed the partial indices of SDI and ADI, Precision and Recall. The various metrics were evaluated for their ability to represent error and their suitability for use in model improvement frameworks.