Sensitivity of measuring the progress in financial risk protection to survey design and its socioeconomic and demographic determinants: A case study in Rwanda

- Survey design had significant effects on estimating the level and trend of the HCHE.
- Poorest households' HCHE status were more likely to change as survey design varied.
- Developing statistical tools was necessary to mitigate survey effects.

Reliable and comparable information on households with catastrophic health expenditure (HCHE) is crucial for monitoring and evaluating our progress towards achieving universal financial risk protection. This study aims to investigate the sensitivity of measuring the progress in financial risk protection to survey design and its socioeconomic and demographic determinants. Using the Rwanda Integrated Living Conditions Survey in 2005 and 2010/2011, we derived the level and trend of the percentage of the HCHE using out-of-pocket health spending data derived from (1) a health module with a two-week recall period and six (2005)/seven (2010/2011) survey questions (Method 1) and (2) a consumption module with a four-week/ten-/12-month recall period and 11(2005)/24 (2010/2011) questions (Method 2). Using multilevel logistic regression analysis, we investigated the household socioeconomic and demographic characteristics that affected the sensitivity of estimating the HCHE to survey design. We found that Method 1 generated a significantly higher HCHE estimate (9.2%, 95% confidence interval 8.4%-10.0%) than Method2 (7.4%, 6.6%-8.1%) in 2005 and lower estimate (5.6%, 5.2%-6.1%) than Method 2 (8.2%, 7.6%-8.7%) in 2010/2011. The estimated trends of the HCHE using the two methods were not consistent between the two years. A household's size, its income quintile, having no under-five children, and educational level of its head were positively associated with the consistency of its HCHE status when using the two survey methods. Estimates of the progress in financial risk protection, especially among the most vulnerable households, are sensitive to survey design. These results are robust to various thresholds of catastrophic health spending. Future work must focus on mitigating survey effects through the development of statistical tools.