On the transition of genetic differentiation from isolation to panmixia: What we can learn from GSTGST and DD

Population genetic differentiation characterizes the repartition of alleles among populations. It is commonly thought that genetic differentiation measures, such as GSTGST and DD, should be near zero when allele frequencies are close to their expected value in panmictic populations, and close to one when they are close to their expected value in isolated populations. To analyse those properties, we first derive analytically a reference function ff of known parameters that describes how important features of genetic differentiation (e.g. gene diversity, proportion of private alleles, frequency of the most common allele) are close to their expected panmictic and isolation value. We find that the behaviour of function ff differs according to three distinct mutation regimes defined by the scaled mutation rate and the number of populations. Then, we compare GSTGST and DD to ff, and demonstrate that their signal of differentiation strongly depends on the mutation regime. In particular, we show that DD captures well the variations of genetic diversity when mutation is weak, otherwise it overestimates it when panmixia is not met. GSTGST detects population differentiation when mutation is intermediate but has a low sensitivity to the variations of genetic diversity when mutation is weak. When mutation is strong the domain of sensitivity of both measures are altered. Finally, we also point out the importance of the number of populations on genetic differentiation measures, and provide recommendations for the use of GSTGST and DD.