Computer-based rating method for evaluating multiple visual stimuli on multiple scales

In this research, two joint evaluation rating methods (focus-on-attribute and drag-and-drop) and a separate evaluation rating method (focus-on-stimulus) are proposed for rating multiple visual stimuli with respect to multiple scales. All three interactive methods incorporate a real-time adjusting mechanism, allowing respondents to interactively adjust their ratings. Using the traditional paper-and-pencil method as the basis for comparison, the performance of these computer-based rating methods and the proposed real-time adjusting mechanism were investigated. First, it was found that the computer-based methods not only produced results equivalent to those produced by the paper-and-pencil method, but also improved the data quality by preventing the occurrence of missing values and allowing respondents to adjust their scores. Second, the two joint evaluation methods performed better than the separate evaluation method in terms of usage experience, test–retest reliability, and the likelihood of simulating the daily shopping experience, in which multiple consumer products are often compared simultaneously. Of the two joint evaluation methods, the drag-and-drop method received better subjective evaluations than the focus-on-attribute method. Third, the proposed real-time adjusting mechanism was found to obtain better subjective satisfaction in rating. In summary, for measuring multiple visual stimuli with multiple scales, the computer-based joint evaluation rating methods using a real-time adjusting mechanism were found to be better choices among the measurement tools investigated.