The Reliability of Survey Data: The Effects of Question Content, Context and Form

Vast amounts of survey data are collected each year for many purposes, including the collection of information for government use, public opinion and election surveys, advertising and marketing research, as well as basic research in the social and economic sciences. Because surveys are so widely used, it is essential that we establish a knowledge base about the measurement quality of survey data and the attributes of questions that produce the best data. Science well understands the reliability of multi-item measures, or scales, commonly used in such research. But we know much less about the quality of single-item survey questions. New data resources and modeling strategies now make it possible to assess the reliability or consistency of measurement for these survey measures. Based on statistical models for evaluating reliability of measurement, this project draws upon the insights from the information gathering process in surveys. The results of this research will provide practical suggestions that will improve the quality of survey data, thus improving survey research generally. The project will develop a publicly available website to allow all survey researchers to investigate question reliability of a large number of commonly used survey questions. Because surveys are used broadly by science, government and the private sector to inform many policy and economic decisions, improved information about survey measures will contribute to the health and well being of U. S. society across multiple sectors and over time.

Because surveys are so widely used, it is essential that we establish a knowledge base about the measurement quality of survey data and the attributes of questions that produce the best results. In order to accomplish these aims, this project develops a set of meta-data containing information for roughly 1200 single-item questions representative of typical questions used in social science surveys. The database of survey questions, developed on the basis of ten nationally (or regionally) representative panel studies including the General Social Surveys, the Health and Retirement Study, the National Election Studies, contains estimates of question-specific reliabilities, along with detailed coding of attributes of the questions, such as their content, response formats, and question length, which can be used to evaluate the optimal properties of survey questions with respect to levels of measurement error. Through an analysis of the reliability information and the attributes of survey questions from these large-scale panel studies, the study focuses on three major elements of questionnaire construction: content, context and form. Statistical analysis will involve use of auto-regressive or Quasi-Markov simplex modeling techniques. The resulting meta-data set will be archived with the Inter-university Consortium for Political and Social Research (ICPSR) and other suitable data archives. In addition, the project will post key elements of these data on a publicly available website, along with images of questions and question context involved, in a manner that allows users to search, filter or query the data base in investigating the reliability of types of survey questions of interest. These findings will contribute important information to the multidisciplinary field of survey methodology, which informs scholars in disciplines including sociology, political science, economics, communication, psychology, law, public health, and public policy, as well as actors in the private and governmental sectors.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.