Making the right conclusions based on wrong results and small sample sizes: Interpretation of statistical tests in ecotoxicology

In environmental risk assessments statistical tests are a standard tool to evaluate the significance of effects by pesticides. While it has rarely been assessed how likely it is to detect effects given a specific sample size, it was never analysed how reliable results are if the test preconditions, particularly of parametric tests, are not fulfilled or how likely it is to detect deviations from these preconditions. Therefore, we analyse the performance of a parametric and a non-parametric test using Monte Carlo simulation, focussing on typical data used in ecotoxicological risk assessments. We show that none of the data distributions are normal and that for typical sample sizes of N<20 it is very unlikely to detect deviations from normality. Non-parametric tests performed markedly better than parametric tests, except when data were in fact normally distributed. We finally discuss the impact of using different tests on pesticide risk assessments.