Challenges to estimating vaccine impact using hospitalization data

- Changes in healthcare delivery can bias vaccine impact estimates based on hospitalization data.
- Improvements in primary care that reduce hospitalizations often coincide with vaccine introduction.
- Vaccine benefits may be underestimated if hospitals are operating at full capacity.
- Commonly used adjustments are not always sufficient to control for bias and confounding.
- We suggest ways to strengthen vaccine impact assessments based on hospitalization data.

Because the real-world impact of new vaccines cannot be known before they are implemented in national programs, post-implementation studies at the population level are critical. Studies based on analysis of hospitalization rates of vaccine-preventable outcomes are typically used for this purpose. However, estimates of vaccine impact based on hospitalization data are particularly prone to confounding, as hospitalization rates are tightly linked to changes in the quality, access and use of the healthcare system, which often occur simultaneously with introduction of new vaccines. Here we illustrate how changes in healthcare delivery coincident with vaccine introduction can influence estimates of vaccine impact, using as an example reductions in infant pneumonia hospitalizations after introduction of the 10-valent pneumococcal conjugate vaccine (PCV10) in Brazil. To this end, we explore the effect of changes in several metrics of quality and access to public healthcare on trends in hospitalization rates before (2008-09) and after (2011−12) PCV10 introduction in 2010. Changes in infant pneumonia hospitalization rates following vaccine introduction were significantly associated with concomitant changes in hospital capacity and the fraction of the population using public hospitals. Importantly, reduction of pneumonia hospitalization rates after PCV10 were also associated with the expansion of outpatient services in several Brazilian states, falling more sharply where primary care coverage and the number of health units offering basic and emergency care increased more. We show that adjustments for unrelated (non-vaccine) trends commonly employed by impact studies, such as use of single control outcomes, are not always sufficient for accurate impact assessment. We discuss several ways to identify and overcome such biases, including sensitivity analyses using different denominators to calculate hospitalizations rates and methods that track changes in the outpatient setting. Employing these practices can improve the accuracy of vaccine impact estimates, particularly in evolving healthcare settings typical of low- and middle-income countries.