Likelihood ratio tests for equality of shape under varying degrees of orientation invariance

We consider a problem from image cytometry where the objective is to describe possible changes in the shape and orientation of cellular nuclei after treatment with a toxin. The shapes of nuclei are represented by individual ellipses. It is argued that the shape comparison problem can be formulated as a generalization of a hypothesis test for the equality of covariance matrices. For many cell types, the test statistic should be invariant with respect to orientations of the cells. For other cell types, the test statistic should be equivariant with respect to orientations of the cells, but invariant with respect to orientations of the images. Likelihood ratio tests (LRTs) are derived under a Wishart model. The likelihood maximization uses a new result about the minimization of the determinant of a sum of matrices under individual rotations. The applicability and limitations of these LRTs are demonstrated by means of simulation experiments. The reference distributions of the test statistics under the null hypothesis are obtained using unrestricted and restricted randomization procedures. Justification for the Wishart model is provided using a residual diagnostic method. The scientific implications of the results are considered.