Selective correlations; not voodoo

- Estimation after selection (“Double Dipping”) made possible with existing statistical tools.
- Solution via selective confidence intervals with proven coverage guarantees.
- Data splits should not be recommended due to low power.
- New “Confidence Calibration Plot” proposed for visualizing interval estimates.

The problem of “voodoo” correlations-exceptionally high observed correlations in selected regions of the brain-is well recognized in neuroimaging. It arises when quantities of interest are estimated from the same data that was used to select them as interesting. In statistical terminology, the problem of inference following selection from the same data is that of selective inference. Motivated by the unwelcome side-effects of splitting the data- the recommended remedy-we adapt the recent developments in selective inference in order to construct confidence intervals (CIs) with good reproducibility prospects, even if selection and estimation are done with the same data. These intervals control the expected proportion of non-covered correlations in the selected voxels-the False Coverage Rate (FCR). They extend further toward zero than standard intervals, thus attenuating the impression made by highly biased observed correlations. They do so adaptively, in that they coincide with the standard CIs when far away from the selection point. We complement existing analytic proofs with a simulation, showing that the proposed intervals control the FCR in realistic social neuroscience problems. We also suggest a “confidence calibration plot”, to allow the intervals to be reported in a clear and interpretable way. Applying the proposed methodology on a loss-aversion study, we demonstrate that with the sample size and selection type employed, selection bias is considerable. Finally, selective intervals are compared to the currently recommended data-splitting approach. We discover that our approach has more power and typically more informative, as no data is discarded.Computation of the intervals is implemented in an accompanying software package.