Towards testing the unitarity of the 3×3 lepton flavor mixing matrix in a precision reactor antineutrino oscillation experiment

The 3×3 Maki-Nakagawa-Sakata-Pontecorvo (MNSP) lepton flavor mixing matrix may be slightly non-unitary if the three active neutrinos are coupled with sterile neutrinos. We show that it is in principle possible to test whether the relation |Ve1|2+|Ve2|2+|Ve3|2=1 holds or not in a precision reactor antineutrino oscillation experiment, such as the recently proposed Daya Bay II experiment. We explore three categories of non-unitary effects on the 3×3 MNSP matrix: 1) the indirect effect in the (3+3) flavor mixing scenario where the three heavy sterile neutrinos do not take part in neutrino oscillations; 2) the direct effect in the (3+1) scenario where the light sterile neutrino can oscillate into the active ones; and 3) the interplay of both of them in the (3+1+2) scenario. We find that both the zero-distance effect and flavor mixing factors of different oscillation modes can be used to determine or constrain the sum of |Ve1|2, |Ve2|2 and |Ve3|2 and its possible deviation from one, and the active neutrino mixing angles θ12 and θ13 can be cleanly extracted even in the presence of light or heavy sterile neutrinos. Some useful analytical results are obtained for each of the three scenarios.