Benchmarking of Localization Solutions: Guidelines for the Selection of Evaluation Points

Indoor localization solutions are key enablers for next-generation indoor navigation and track and tracing solutions. As a result, an increasing number of different localization algorithms have been proposed and evaluated in scientific literature. However, many of these publications do not accurately substantiate the used evaluation methods. In particular, many authors utilize a different number of evaluation points, but they do not (i) analyze if the number of used evaluation points is sufficient to accurately evaluate the performance of their solutions and (ii) report on the uncertainty of the published results. To remedy this, this paper evaluates the influence of the selection of evaluation points. Based on statistical parameters such as the standard error of the mean value, an estimator is defined that can be used to quantitatively analyze the impact of the number of used evaluation points on the confidence interval of the mean value of the obtained results. This estimator is used to estimate the uncertainty of the presented accuracy results, and can be used to identify if more evaluations are required. To validate the proposed estimator, two different localization algorithms are evaluated in different testbeds and using different types of technology, showing that the number of required evaluation points does indeed vary significantly depending on the evaluated solution.