Original research paperA comparison of methods for the analysis of binomial clustered outcomes in behavioral research

- We performed a comparison of statistical methods for the analysis of clustered binary outcomes in behavioral research with small sample sizes.
- Beta-binomial regression performed accurate and powerful hypothesis testing, outperforming even Generalized Linear Mixed Models in a range of scenarios.
- A misspecified linear model, in some circumstances, can represent a reasonable compromise between technical approachability and accuracy when dealing with proportion data.
- Poisson regression should not be applied straight away to modeling of proportion data.

BackgroundIn behavioral research, data consisting of a per-subject proportion of “successes” and “failures” over a finite number of trials often arise. This clustered binary data are usually non-normally distributed, which can distort inference if the usual general linear model is applied and sample size is small. A number of more advanced methods is available, but they are often technically challenging and a comparative assessment of their performances in behavioral setups has not been performed.MethodWe studied the performances of some methods applicable to the analysis of proportions; namely linear regression, Poisson regression, beta-binomial regression and Generalized Linear Mixed Models (GLMMs). We report on a simulation study evaluating power and Type I error rate of these models in hypothetical scenarios met by behavioral researchers; plus, we describe results from the application of these methods on data from real experiments.ResultsOur results show that, while GLMMs are powerful instruments for the analysis of clustered binary outcomes, beta-binomial regression can outperform them in a range of scenarios. Linear regression gave results consistent with the nominal level of significance, but was overall less powerful. Poisson regression, instead, mostly led to anticonservative inference.Comparison with existing methodsGLMMs and beta-binomial regression are generally more powerful than linear regression; yet linear regression is robust to model misspecification in some conditions, whereas Poisson regression suffers heavily from violations of the assumptions when used to model proportion data.ConclusionsWe conclude providing directions to behavioral scientists dealing with clustered binary data and small sample sizes.