Unconventional superconducting states on doped Shastry–Sutherland lattice

•We investigate the strong correlated SS lattice by using Gutziwiller projection operator and mean-field theory.•In the most frustrated point η=±1,η=±1, various state appears, the energy favorable state is the ground state.•There are four distinct phases, two superconducting phase and two none-superconducting phases.•The order of phase transition between different phases depends on the continuity characteristics of order parameters.

By using a renormalized mean-field theory, we investigate the phase diagram of t–t′–J–J′ model on a two dimensional Shastry–Sutherland (SS) lattice which can be realized in a group of layered compounds such as SrCu2(BO3)2, Yb2Pt2Pb. We find that the symmetry of ground state depends on frustration parameter η=t′/tη=t′/t and doping concentration. For weak and intermediate |η  |, dx2−y2dx2−y2-wave superconducting state is robust in a wide region. While for larger |η| > 1 cases, the superconducting ground state has s–s  -wave symmetry. Around the most frustrated η=±1η=±1 point, dx2−y2dx2−y2-wave state, s–s-wave state as well as staggered-flux state appear, the energy favorable state is the ground state. The order of the phase transition between different states depends on the continuity of the mean field order parameters.