Measure for data partitioning in m × 2 cross-validation

An m × 2 cross-validation based on m half-half partitions is widely used in machine learning. However, the cross-validation performance often relies on the quality of the data partitioning. Poor data partitioning may cause poor inference results, such as a large variance and large Type I and II errors of the corresponding test. To evaluate the quality of the data partitioning, we propose a statistic based on the difference between the observed and expected numbers of overlapped samples of two training sets in an m × 2 cross-validation. The expectation and variance of the proposed statistic are also given. Furthermore, by studying the quantile of the distribution of the statistic, we find that the occurrence of poor data partitioning is not a small probability event. Thus, data partitioning should be predesigned before conducting m × 2 cross-validation experiments in machine learning such that the number of overlapped samples observed is equal or as close as possible to the number expected.