How Not to Do WLS Fitting in Calibration with Heteroscedastic Data

In the least-squares fitting of data of varying precision to functions that are algebraically linear in the adjustable parameters, the correct weights for obtaining optimal results are the pointwise inverse variances σi-2. These have often been obtained from the sampling statistics of replicate measurements of the uncertain quantity y, a procedure that has come to define weighted least-squares (WLS) calibration fitting. Unfortunately, such variance estimates are notoriously imprecise, and in a long-ago Monte Carlo study involving a linear response function and weights ranging over a factor of 5, ordinary least squares (OLS) outperformed WLS for almost every tested set of calibration x values and replicates. But there is a better way of obtaining the weights: variance-function estimation. This method relies on the fact that the variances for most physical data follow simple, smooth functions having two or three terms proportional to a constant, y, and y2. Since the replicate data are used collectively to estimate the variance function, the number of statistical degrees of freedom is large enough to achieve good precision in the calibration weights. When variance-function weighting is included in the aforementioned OLS-vs-WLS comparisons, it wins over OLS for almost every combination with m = 4-8 x values and n = 3-5 replicates. Further, of the four ways of weighting in the variance-function fitting, the iterative reweighting method gives the best results, in support of earlier findings.