Assessing the ‘Gold Standard’ of party policy placements: Is computerized replication possible?

The Comparative Manifestos Project (CMP) offers data on party policy positions based on a coding scheme of 56 categories. To what extent can we replicate the CMP coding results by using computerized topical coding of digitalised manifestos? Both human and computer coding have a number of strengths and weaknesses so that the combination of strengths could help to arrive at both valid and reliable party policy placements, in particular the measurement of policy movements over time. More than 1500 digitalised party manifestos in 20 democracies in the period 1960–2009 are re-coded with a computerized CMP-compatible coding scheme in order to assess the differences with human coding using exactly the same issue categories and the same left–right scale.The analysis shows that it is possible to use computer coding in order to locate the potential weakness of the human coding and the other way around. It also illustrates that the validity and reliability of policy placements is a function of the conceptualisation and operationalisation of issues, of the size of documents and of scale construction. Computerized cross-validation of the CMP-coding results offers a new and powerful tool to assess its reliability.

► Computerized replication helps to assess the reliability of estimated policy change. ► The success of computerized replication differs per issue. ► The best performance is achieved in one-dimensional categories. ► The performance also depends on proper document collection and scale construction.