Backfitting and local likelihood methods for nonparametric mixed-effects models with longitudinal data

For longitudinal data analyses, it is important to estimate both population (mean) response and subject-specific individual responses. We consider a nonparametric mixed-effects model that characterizes both population effect and random effects as nonparametric functions, i.e., yi(tij)=η(tij)+υi(tij)+εi(tij),i=1,…,m,j=1,…,ni. Although similar models have been studied, in this paper we propose a novel approach to use the local likelihood concept and a backfitting algorithm to combine the local polynomial regression techniques and the linear mixed-effects (LME) model methods for efficiently estimating both population mean and individual curve functions. The asymptotic properties of the proposed estimators are established for two scenarios: (i) the number of subjects (mm) and the number of measurements per subject (nini) tend to infinity; and (ii) mm tends to infinity while nini is finite. The simulation studies are carried out to compare the performance of our proposed estimator with that of an existing estimator, local polynomial LME (LLME) estimator proposed by Wu and Zhang [J. Amer. Statist. Assoc. 97 (2002) 883–897], and we show that our approach performs better than the existing method in the sense of mean-squared errors (MSE). We illustrate our estimation method with an application to an AIDS clinical study.