Cell-specific and post-hoc spatial clustering tests based on nearest neighbor contingency tables

Spatial clustering patterns in a multi-class setting such as segregation and association between classes have important implications in various fields, e.g., in ecology, and can be tested using nearest neighbor contingency tables (NNCTs). A NNCT is constructed based on the types of the nearest neighbor (NN) pairs and their frequencies. We survey the cell-specific (or pairwise) and overall segregation tests based on NNCTs in literature and introduce new ones and determine their asymptotic distributions. We demonstrate that cell-specific tests enjoy asymptotic normality, while overall tests have chi-square distributions asymptotically. Some of the overall tests are confounded by the unstable generalized inverse of the rank-deficient covariance matrix. To overcome this problem, we propose rank-based corrections for the overall tests to stabilize their behavior. We also perform an extensive Monte Carlo simulation study to compare the finite sample performance of the tests in terms of empirical size and power based on the asymptotic and Monte Carlo critical values and determine the tests that have the best size and power performance and are robust to differences in relative abundances (of the classes). In addition to the cell-specific tests, we discuss one(-class)-versus-rest type of tests as post-hoc tests after a significant overall test. We also introduce the concepts of total, strong, and partial segregation/association to differentiate different levels of these patterns. We compare the new tests with the existing NNCT-tests in literature with simulations and illustrate the tests on an ecological data set.