Survey data collection methods and discrepancy in the sociological study of religious congregations

Surveys of religious congregations are a mainstay of sociological research on organized religion in the United States. How accurate, reliable, and comparable are the data generated from the disparate methods used by researchers? We analyze four congregational surveys to show how two components of data collection-sampling design and survey response rate-may contribute to differences in population estimates between the surveys. Results show that in three populations of congregations (all religious traditions, Catholic parishes, and Hispanic Catholic parishes), estimates of key congregational measures, such as head clergy characteristics, congregational size, and Hispanic composition, are susceptible to differences in data collection methods. While differences in sampling design contribute to some of the variation in variable estimates, our unique analysis of survey metadata shows the importance of high response rates for producing accurate estimates for many variables. We conclude with suggestions for improving congregational data collection methods and efforts to compare survey estimates.