Scaled Paired Comparisons as an Alternative to Ratings and Rankings for Measuring Values

This paper introduces and assesses a method for estimating individual and group relative preferences of values. The method of scaled paired comparisons is introduced in the context of prior research on rating methods struggles to avoid nondifferentiation when rating sets of popular objects that constitute a value system. The method of scaled paired comparisons is introduced, along with a planned missing procedure that allows the method to be applied to large number of values. Using data from a survey experiment, the paper compares scaled paired comparisons with rating scales modeled on those used in the European Social Survey. This exploratory study finds that scaled paired comparisons require more time to complete but provide measures with greater reliability and higher construct validity than rating questions using identical question stems. While the particular example in this paper concerns liberal democracy, the method's foundations lie in the study of terminal and instrumental values, and the method has the potential to improve measurement of those value systems and others such as post-materialism, child rearing, and more. More generally, it may be an attractive substitute for rating scales whenever the objects being rated are sufficiently popular (or unpopular) as to induce nondifferentiation and valid straightlining.