New adaptive strategies for nonparametric estimation in linear mixed models

•We focus on non-parametric estimation in linear mixed models.•We propose two new nonparametric estimators of the density of the random effect.•Final estimators are built with adaptive selection methods and theoretical properties are studied.•The performances of the estimators are illustrated on a simulation study.

This paper surveys new estimators of the density of a random effect in linear mixed-effects models. Data are contaminated by random noise, and we do not observe directly the random effect of interest. The density of the noise is supposed to be known, without assumption on its regularity. However it can also be estimated. We first propose an adaptive nonparametric deconvolution estimation based on a selection method set up in Goldenshluger and Lepski (2011). Then we propose an estimator based on a simpler model selection deviced by contrast penalization. For both of them, non-asymptotic L2-riskL2-risk bounds are established implying estimation rates, much better than the expected deconvolution ones. Finally the two data-driven strategies are evaluated on simulations and compared with previous proposals.