Investigating variation in replicability: A "many labs" replication project

Richard A. Klein; Kate A. Ratliff; Michelangelo Vianello; Reginald B. Adams; Štěpán Bahník; Michael J. Bernstein; Konrad Bocian; Mark J. Brandt; Beach Brooks; Claudia Chloe Brumbaugh; Zeynep Cemalcilar; Jesse Chandler; Winnee Cheong; William E. Davis; Thierry Devos; Matthew Eisner; Natalia Frankowska; David Furrow; Elisa Maria Galliani; Fred Hasselman; Joshua A. Hicks; James F. Hovermale; S. Jane Hunt; Jeffrey R. Huntsinger; Hans Ijzerman; Melissa Sue John; Jennifer A. Joy-Gaba; Heather Barry Kappes; Lacy E. Krueger; Jaime Kurtz; Carmel A. Levitan; Robyn K. Mallett; Wendy L. Morris; Anthony J. Nelson; Jason A. Nier; Grant Packard; Ronaldo Pilati; Abraham M. Rutchick; Kathleen Schmidt; Jeanine L. Skorinko; Robert Smith; Troy G. Steiner; Justin Storbeck; Lyn M. Van Swol; Donna Thompson; A. E. Van 'T Veer; Leigh Ann Vaughn; Marek Vranka; Aaron L. Wichman; Julie A. Woodzicka; Brian A. Nosek

doi:10.1027/1864-9335/a000178

Investigating variation in replicability: A "many labs" replication project

Richard A. Klein
, Kate A. Ratliff
, Michelangelo Vianello
, Reginald B. Adams
, Štěpán Bahník
, Michael J. Bernstein
, Konrad Bocian
, Mark J. Brandt
, Beach Brooks
, Claudia Chloe Brumbaugh
, Zeynep Cemalcilar
, Jesse Chandler
, Winnee Cheong
, William E. Davis
, Thierry Devos
, Matthew Eisner
, Natalia Frankowska
, David Furrow
, Elisa Maria Galliani
, Fred Hasselman

Joshua A. Hicks, James F. Hovermale, S. Jane Hunt, Jeffrey R. Huntsinger, Hans Ijzerman, Melissa Sue John, Jennifer A. Joy-Gaba, Heather Barry Kappes, Lacy E. Krueger, Jaime Kurtz, Carmel A. Levitan, Robyn K. Mallett, Wendy L. Morris, Anthony J. Nelson, Jason A. Nier, Grant Packard, Ronaldo Pilati, Abraham M. Rutchick, Kathleen Schmidt, Jeanine L. Skorinko, Robert Smith, Troy G. Steiner, Justin Storbeck, Lyn M. Van Swol, Donna Thompson, A. E. Van 'T Veer, Leigh Ann Vaughn, Marek Vranka, Aaron L. Wichman, Julie A. Woodzicka, Brian A. Nosek

Research output: Contribution to journal › Article › peer-review

853 Scopus citations

Abstract

Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect - imagined contact reducing prejudice - showed weak support for replicability. And two effects - flag priming influencing conservatism and currency priming influencing system justification - did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.

Original language	English (US)
Pages (from-to)	142-152
Number of pages	11
Journal	Social Psychology
Volume	45
Issue number	3
DOIs	https://doi.org/10.1027/1864-9335/a000178
State	Published - 2014

All Science Journal Classification (ASJC) codes

Social Psychology
Arts and Humanities (miscellaneous)
Sociology and Political Science
General Psychology

Access to Document

10.1027/1864-9335/a000178

Cite this

Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., ... Nosek, B. A. (2014). Investigating variation in replicability: A "many labs" replication project. Social Psychology, 45(3), 142-152. https://doi.org/10.1027/1864-9335/a000178

@article{6ec44fad54a943dcace8ba7287b1fe9c,

title = "Investigating variation in replicability: A {"}many labs{"} replication project",

abstract = "Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect - imagined contact reducing prejudice - showed weak support for replicability. And two effects - flag priming influencing conservatism and currency priming influencing system justification - did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.",

author = "Klein, \{Richard A.\} and Ratliff, \{Kate A.\} and Michelangelo Vianello and Adams, \{Reginald B.\} and {\v S}t{\v e}p{\'a}n Bahn{\'i}k and Bernstein, \{Michael J.\} and Konrad Bocian and Brandt, \{Mark J.\} and Beach Brooks and Brumbaugh, \{Claudia Chloe\} and Zeynep Cemalcilar and Jesse Chandler and Winnee Cheong and Davis, \{William E.\} and Thierry Devos and Matthew Eisner and Natalia Frankowska and David Furrow and Galliani, \{Elisa Maria\} and Fred Hasselman and Hicks, \{Joshua A.\} and Hovermale, \{James F.\} and Hunt, \{S. Jane\} and Huntsinger, \{Jeffrey R.\} and Hans Ijzerman and John, \{Melissa Sue\} and Joy-Gaba, \{Jennifer A.\} and Kappes, \{Heather Barry\} and Krueger, \{Lacy E.\} and Jaime Kurtz and Levitan, \{Carmel A.\} and Mallett, \{Robyn K.\} and Morris, \{Wendy L.\} and Nelson, \{Anthony J.\} and Nier, \{Jason A.\} and Grant Packard and Ronaldo Pilati and Rutchick, \{Abraham M.\} and Kathleen Schmidt and Skorinko, \{Jeanine L.\} and Robert Smith and Steiner, \{Troy G.\} and Justin Storbeck and \{Van Swol\}, \{Lyn M.\} and Donna Thompson and \{Van 'T Veer\}, \{A. E.\} and Vaughn, \{Leigh Ann\} and Marek Vranka and Wichman, \{Aaron L.\} and Woodzicka, \{Julie A.\} and Nosek, \{Brian A.\}",

year = "2014",

doi = "10.1027/1864-9335/a000178",

language = "English (US)",

volume = "45",

pages = "142--152",

journal = "Social Psychology",

issn = "1864-9335",

publisher = "Hogrefe Publishing GmbH",

number = "3",

}

Klein, RA, Ratliff, KA, Vianello, M, Adams, RB, Bahník, Š, Bernstein, MJ, Bocian, K, Brandt, MJ, Brooks, B, Brumbaugh, CC, Cemalcilar, Z, Chandler, J, Cheong, W, Davis, WE, Devos, T, Eisner, M, Frankowska, N, Furrow, D, Galliani, EM, Hasselman, F, Hicks, JA, Hovermale, JF, Hunt, SJ, Huntsinger, JR, Ijzerman, H, John, MS, Joy-Gaba, JA, Kappes, HB, Krueger, LE, Kurtz, J, Levitan, CA, Mallett, RK, Morris, WL, Nelson, AJ, Nier, JA, Packard, G, Pilati, R, Rutchick, AM, Schmidt, K, Skorinko, JL, Smith, R, Steiner, TG, Storbeck, J, Van Swol, LM, Thompson, D, Van 'T Veer, AE, Vaughn, LA, Vranka, M, Wichman, AL, Woodzicka, JA & Nosek, BA 2014, 'Investigating variation in replicability: A "many labs" replication project', Social Psychology, vol. 45, no. 3, pp. 142-152. https://doi.org/10.1027/1864-9335/a000178

TY - JOUR

T1 - Investigating variation in replicability

T2 - A "many labs" replication project

AU - Klein, Richard A.

AU - Ratliff, Kate A.

AU - Vianello, Michelangelo

AU - Adams, Reginald B.

AU - Bahník, Štěpán

AU - Bernstein, Michael J.

AU - Bocian, Konrad

AU - Brandt, Mark J.

AU - Brooks, Beach

AU - Brumbaugh, Claudia Chloe

AU - Cemalcilar, Zeynep

AU - Chandler, Jesse

AU - Cheong, Winnee

AU - Davis, William E.

AU - Devos, Thierry

AU - Eisner, Matthew

AU - Frankowska, Natalia

AU - Furrow, David

AU - Galliani, Elisa Maria

AU - Hasselman, Fred

AU - Hicks, Joshua A.

AU - Hovermale, James F.

AU - Hunt, S. Jane

AU - Huntsinger, Jeffrey R.

AU - Ijzerman, Hans

AU - John, Melissa Sue

AU - Joy-Gaba, Jennifer A.

AU - Kappes, Heather Barry

AU - Krueger, Lacy E.

AU - Kurtz, Jaime

AU - Levitan, Carmel A.

AU - Mallett, Robyn K.

AU - Morris, Wendy L.

AU - Nelson, Anthony J.

AU - Nier, Jason A.

AU - Packard, Grant

AU - Pilati, Ronaldo

AU - Rutchick, Abraham M.

AU - Schmidt, Kathleen

AU - Skorinko, Jeanine L.

AU - Smith, Robert

AU - Steiner, Troy G.

AU - Storbeck, Justin

AU - Van Swol, Lyn M.

AU - Thompson, Donna

AU - Van 'T Veer, A. E.

AU - Vaughn, Leigh Ann

AU - Vranka, Marek

AU - Wichman, Aaron L.

AU - Woodzicka, Julie A.

AU - Nosek, Brian A.

PY - 2014

Y1 - 2014

N2 - Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect - imagined contact reducing prejudice - showed weak support for replicability. And two effects - flag priming influencing conservatism and currency priming influencing system justification - did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.

AB - Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect - imagined contact reducing prejudice - showed weak support for replicability. And two effects - flag priming influencing conservatism and currency priming influencing system justification - did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.

UR - https://www.scopus.com/pages/publications/84901716350

UR - https://www.scopus.com/pages/publications/84901716350#tab=citedBy

U2 - 10.1027/1864-9335/a000178

DO - 10.1027/1864-9335/a000178

M3 - Article

SN - 1864-9335

VL - 45

SP - 142

EP - 152

JO - Social Psychology

JF - Social Psychology

IS - 3

ER -

Investigating variation in replicability: A "many labs" replication project

Abstract

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this