Performance evaluation of output analysis methods in steady-state simulations

Output analysis methods of steady-state simulations have extensively been subject of study to evaluate the performance when estimating the mean. However, smaller efforts have been placed on performance evaluation of these methods to estimate variance and quantiles. In this paper, we empirically evaluate the performance of output analysis methods based on multiple replications and batches to estimate mean, variance and quantile with the same set of data. The evaluation of the performance of the methods is based on the empirical coverage of the true value using confidence intervals, the average bias, relative error and mean squared error. The methods are applied to estimate the average, variance and quantiles of waiting time in an M/M/1M/M/1 queue. The results show that the methods based on non-overlapping batches perform consistently well in all the metrics. The performance of the other methods varies depending on the metric and the parameters of the simulation. In addition, we provide another example of a non-geometric ergodic Markov chain to show that asymptotically valid confidence intervals for quantiles can be obtained using batches and replications.