Research note: An evaluation of the Elo algorithm for pairwise visual assessment surveys

A growing number of online visual preference surveys are conducted which ask respondents to make pairwise comparisons between visual stimuli, raising questions about how to best analyze the resulting data. This research note directly compares three potential methods for analyzing such data, two which have been previously described (Win Ratio and Q Score) and one newly applied to this context (Elo Score). Drawing on a dataset of 103,200 votes among 362 randomly selected streets in Philadelphia, the note finds that although the three metrics are highly correlated overall, the Elo Score results in more dispersed standardized scores and a somewhat different rank order than the Win Ratio (Kendall's tau-b 0.864). Although the Win Ratio may be adequate for some research designs, the Elo Score provides a more discriminating measure for images that elicit strong responses. The note discusses how studies using pairwise comparisons might incorporate respondent demographics, and recommends further investigation of respondent demographics for online preference surveys.