Least-squares analysis of data with uncertainty in x and y: A Monte Carlo methods comparison

The least-squares analysis of data with error in x and y is generally thought to yield best results when the quantity minimized is the sum of the properly weighted squared residuals in x and in y. As an alternative to this “total variance” (TV) method, “effective variance” (EV) methods convert the uncertainty in x into an effective contribution to that in y, and though easier to use are considered to be less reliable. There are at least two EV methods, differing in how the weights are treated in the optimization. One of these is identical to the TV method for fits to a straight line. The formal differences among these methods are clarified, and Monte Carlo simulations are used to examine the statistical properties of each on the widely used straight-line model of York, a quadratic variation on this, Orear's hyperbolic model, a nonlinear binding (Langmuir) model, and Wentworth's kinetics model. The simulations confirm that the EV and TV methods are statistically equivalent in the limit of small data error, where they yield unbiased, normally distributed parameter estimates, with standard errors correctly predicted by the a priori covariance matrix. With increasing data error, these properties fail to hold; and the TV method is not always statistically best. Nonetheless, the method differences should seldom be of practical significance, since they are likely to be small compared with uncertainties from incomplete information about the data error in x and y.