Generalizability of songbird habitat models in boreal mixedwood forests of Alberta

Statistical models relating habitat characteristics to species occurrences are increasingly used to evaluate the consequences of forest management activities and conservation plans over large spatial and temporal scales. In practice, such models do not always generalize to other locations, hence, they should be validated using independent data. In this paper, we assess the predictive ability of 16 songbird habitat models developed in the Calling Lake region of the boreal mixedwood forest of Alberta using both internal and external validation approaches. Internal validation relied on the same dataset used to develop the models while external validation utilised independent data collected within four validation landscapes in the same ecological region. Two aspects of predictive accuracy were evaluated: the agreement between observations and predicted values (calibration) and the models’ ability to classify locations into those in which species are present or absent (discrimination). Calibration was assessed using the Hosmer–Lemeshow (H–L) statistic and plots showing predicted versus observed probabilities of occurrence. Discrimination was assessed using receiver operating characteristic (ROC) curves and the associated area under the ROC curve.With internal validation, the calibration plots were reasonable for all species, however the H–L statistic indicated a good fit for only eight species. Model discrimination between occupied and unoccupied sites, on the other hand, was reasonable for 14 of 16 models. With external validation, both the calibration plots and the H–L statistic identified only three species with a reasonable fit. The remaining models generally over- or under-predicted probability of occurrence. External model discrimination was reasonable for 10 of 16 species. When we re-estimated the models using all available data, the calibration plots were again reasonable for all species, with the H–L statistic indicating good fit for 12 species. Similarly, model discrimination was reasonable for 14 of 16 species.Several factors may help to explain why model performance using external data was generally poorer and more variable than when using internal data. These include differences in landscape structure and disturbance history as well as differences in the frequency of occurrence of individual species. Overall, our analyses identified several species whose habitat models degraded considerably, especially as measured by calibration. By re-estimating the models, we increased the range of variation in our covariates which likely led to an improvement in model calibration. We discuss the importance of ongoing evaluation and refinement of habitat models that will be used in conservation planning or to evaluate management scenarios at large spatial and temporal scales.