Confronting problems of method in the study of sustainability

• Forest policies depend on actionable, reliable and valid research findings.
• Problems of method remain an important issue for research on forest commons.
• Categorical data analysis are particularly problematic in small samples.
• Monte Carlo simulations reveals that this problem is somewhat overstated.
• A multi-method research agenda is best suited to produce strong policy advice.

Methodological challenges have confounded studies of the commons for much of its recent history. These problems range from a large number of potentially influential variables, difficulties in capturing and measuring these variables, and large sets of interacting and mediating factors. With respect to quantitative methods, a specific dilemma is their use at or beyond their methodologically accepted limits. More specifically, this paper explores the potential implications of applying multinomial logistic regression techniques in small samples. It does so by drawing upon published data from the International Forestry Resources and Institutions (IFRI) program to explore potential problems of method from prominent research findings in the literature on forest commons. A subset of the results is then compared to a similar study of tropical deforestation. The results point to considerable inconsistency in the sign of parameter estimates, and a large number of type II errors. However, they also suggest that type I errors are relatively rare. As a whole, this paper demonstrates the general reliability of multinomial logistic regression in small samples by showing that statistically significant parameters are unlikely to lead policymakers in the wrong direction. Nevertheless it also suggests that this approach is likely to overlook several influential factors, posing potential dilemmas for the development of a theory of sustainability.