The choice between different statistical approaches to risk-adjustment influenced the identification of outliers

ObjectiveMany statistical approaches have been applied to compare health care providers' performance, but few studies have examined how the outlier status of providers depends on the choice between risk-adjustment techniques.Study Design and SettingWe analyzed the recourse to breast-conserving surgery (BCS) for breast carcinoma across 31 hospitals of the Veneto Region (Italy). The following methods were compared: the ratio of observed to expected events (O/E), regression models with provider effects introduced as dummy variables obtained by standard or weighted effect coding, and multilevel analysis.ResultsThe O/E method classified seven hospitals (one with high and six with low BCS rates) as outliers. The regression model with the weighted parameterization gave similar results, whereas through standard effect coding all odds ratios shifted and different outliers were identified. Multilevel analysis was quite conservative in identifying small hospitals with BCS rates lower than the regional mean.ConclusionWhenever feasible, results obtained through different statistical methodologies should be compared. If providers are modeled as dummy variables obtained by effect coding, departures of the model intercept from the regional mean should be checked. The increasing use of multilevel models could entail a lower sensitivity in identifying low-quality outliers if a volume–outcome relationship exists.