Frustrated three dimensional anisotropic antiferromagnet with single ion anisotropy

•The quantum phase transitions of the model were studied.•The effect of competing interaction was analyzed.•The phase diagram at zero temperature was presented.

The SU(3) Schwinger bosons formalism is used to study the three dimensional anisotropic Heisenberg antiferromagnet with nearest and next-nearest neighbor interactions in the XY plane (and only nearest neighbor interactions along the z direction) and single ion anisotropy. The quantum phase transitions from the gapped disordered state to the ordered phases are studied. The phase diagram at zero temperature is calculated. We have two kinds of quantum phase transitions: one induced by the single-ion anisotropy term, and another caused by competitive interactions.