Better than you think: Interval estimators of the difference of binomial proportions

• Compares coverage probabilities and widths of 5 explicitly defined nominal 95% confidence intervals for the risk difference.
• Colored contour plots give overviews of coverage and widths to enable visual comparisons.
• Scoring functions make exact the notion of ‘approximate nominal coverage’ and allow for ranking of interval estimates of the risk difference.
• The mean of a variance stabilized risk difference estimator is shown to closely approximate the Kullback–Leibler symmetrized divergence between null and alternative distributions.

The paper studies explicitly defined interval estimation of the difference in proportions arising from independent binomial distributions for small to moderate sample sizes. In particular, the interval proposed by Agresti and Caffo is compared with the Newcombe interval, the KMS interval of Kulinskaya, Morgenthaler and Staudte, the Wald interval and the ‘Jeffreys’ interval proposed by Brown and Li. Our comparative contour plot summaries empirical studies help to identify where each of the methods performs best in terms of coverage and width. For example, for very unbalanced designs we recommend the Newcombe intervals. For obtaining the nominal coverage, the KMS intervals are recommended, providing coverages nearly always between 95% and 97%. Two new summary scores for interval coverage are introduced. In addition to comprehensive empirical findings, this paper also connects the mean value of the KMS variance stabilized statistic to the Kullback–Leibler symmetrized divergence, which helps to explain the good coverage properties of the interval based on it.