Policy Evaluation via Composite Indexes: Qualitative Lessons from International Transparency Policy Indexes

International transparency policy indexes (ITPIs) help determine billions in investment and aid, influence “authoritative” scholarship, and shape policy choices. Are ITPIs valid yardsticks of transparency, or do they encourage dissimulation? Most scholarship on index-based evaluations focuses on “concept indexes” (e.g., governance) from quantitative approaches. This paper presents qualitative insights about ITPIs in specific and “policy indexes” in general, analyzing three measurement-related pitfalls and proposing countermeasures. Most significantly, it shows how indexes presuppose substitutability while policies contain nonsubstitutable 'necessary' policy provisions. This dilemma of “ontological compatibility” means that policies can rank favorably on indexes notwithstanding the absence of lynchpin policy provisions.