Invariant theory for singular α-determinants

From the irreducible decompositions' point of view, the structure of the cyclic GLn(C)-module generated by the α-determinant degenerates when (1⩽k⩽n−1) (see [S. Matsumoto, M. Wakayama, Alpha-determinant cyclic modules of gln(C), J. Lie Theory 16 (2006) 393–405]). In this paper, we show that -determinant shares similar properties which the ordinary determinant possesses. From this fact, one can define a new (relative) invariant called a wreath determinant. Using (GLm,GLn)-duality in the sense of Howe, we obtain an expression of a wreath determinant by a certain linear combination of the corresponding ordinary minor determinants labeled by suitable rectangular shape tableaux. Also we study a wreath determinant analogue of the Vandermonde determinant, and then, investigate symmetric functions such as Schur functions in the framework of wreath determinants. Moreover, we examine coefficients which we call (n,k)-sign appeared at the linear expression of the wreath determinant in relation with a zonal spherical function of a Young subgroup of the symmetric group Snk.