A two-way analysis of variance model with positive definite interaction for homologous factors

A special type of modelling of interaction is investigated in the framework of two-way analysis of variance models for homologous factors. Factors are said to be homologous when their levels are in a meaningful one-to-one relationship, which arise in a wide variety of contexts, as recalled by McCullagh (J. Roy. Statist. Soc. B 62 (2000) 209). The classical linear context for analysis of interaction is extended by positive definiteness restrictions on the interaction parameters. These restrictions aim to provide a spatial representation of the interaction. Properties of the maximum likelihood estimators are derived for a given dimensionality of the model. When the dimension is unknown, an alternative procedure is proposed based on a penalty approach. This approach relies heavily on random matrix theory arguments but we focus on their statistical consequences especially on the reduction of over-fitting problems in the maximum likelihood estimation. Confidence ellipses are provided for an illustrative example.