Degenerate principal series representations and their holomorphic extensions

Let X=H/L be an irreducible real bounded symmetric domain realized as a real form in an Hermitian symmetric domain D=G/K. The intersection S of the Shilov boundary of D with X defines a distinguished subset of the topological boundary of X and is invariant under H. It can be realized as S=H/P for certain parabolic subgroup P of H. We study the spherical representations of H induced from P. We find formulas for the spherical functions in terms of the Macdonald hypergeometric function. This generalizes the earlier result of Faraut–Koranyi for Hermitian symmetric spaces D. We consider a class of H-invariant integral intertwining operators from the representations on L2(S) to the holomorphic representations of G restricted to H. We construct a new class of complementary series for the groups H=SO(n,m), SU(n,m) (with n−m>2) and Sp(n,m) (with n−m>1). We realize them as discrete components in the branching rule of the analytic continuation of the holomorphic discrete series of G=SU(n,m), SU(n,m)×SU(n,m) and SU(2n,2m) respectively.