Systematic underestimation resulting from measurement error in score-based ecological indices

Many ecological indices convert raw estimates of a variable into a score, and combine scores for multiple variables into a total score. Biodiversity valuation indices, for example, convert raw estimates of variables thought to be surrogates for habitat attributes into a quantitative expression of biodiversity value for use in market-based conservation policies. This study evaluated two potential sources of inaccuracy in total scores: quantitative or model uncertainty; and input data uncertainty or measurement error. Simulated scenarios were run to explore the effects of measurement error and bias on hypothetical sites in a grassy woodland, using two indices of biodiversity value commonly used in Australia. Parameterisation of measurement error was informed by published information on empirical measurement error, as well as results of a field trial using 10 independent observers. As expected, larger observer errors led to less precision in valuations, although coefficients of variation in total scores were often smaller than those of the input variables due to compensatory errors. Unexpectedly, unbiased observer errors in the input variables generated biased valuations of biodiversity. This bias was almost always in one direction – the indices underestimated the true biodiversity value of most sites, high-value sites in particular. Underestimation primarily occurred where raw true values were within benchmark scoring categories but observer error caused a majority of observer estimates to fall into lower scoring categories. The size of benchmark scoring intervals and parameter weightings were the two factors that most influenced the degree of bias. Depending on the index used to calculate biodiversity value, the underestimation effect was more significant for sites with intact woody or herbaceous features, due to scoring intervals and weightings of these features. For market-based biodiversity policies, significant implications include increasing the risk of worse than expected outcomes. This research highlights the imperative to minimise observer error hitherto poorly dealt with, to design and construct quantitative ecological indices in ways that do not exacerbate uncertainty, and to explicitly address uncertainty in decision making.

► Simulations revealed that unbiased observer error may lead to biased biodiversity valuations in benchmarked multi-metric indices.
► Bias was almost always to underestimate the true value of sites, and to a greater degree on higher-value sites.
► Bias was exacerbated where benchmark scoring intervals were narrow and for variables that were heavily weighted.
► Underestimation of biodiversity value may lead to a greater than expected risk of net biodiversity loss in offset decisions.