The influence of hidden researcher decisions in applied microeconomics

Nick Huntington-Klein; Andreu Arenas; Emily Beam; Marco Bertoni; Jeffrey R. Bloem; Pralhad Burli; Naibin Chen; Paul Grieco; Godwin Ekpe; Todd Pugatch; Martin Saavedra; Yaniv Stopnitzky

doi:10.1111/ecin.12992

The influence of hidden researcher decisions in applied microeconomics

Nick Huntington-Klein, Andreu Arenas, Emily Beam, Marco Bertoni, Jeffrey R. Bloem, Pralhad Burli, Naibin Chen, Paul Grieco, Godwin Ekpe, Todd Pugatch, Martin Saavedra, Yaniv Stopnitzky

Economics

Research output: Contribution to journal › Article › peer-review

78 Scopus citations

Abstract

Researchers make hundreds of decisions about data collection, preparation, and analysis in their research. We use a many-analysts approach to measure the extent and impact of these decisions. Two published causal empirical results are replicated by seven replicators each. We find large differences in data preparation and analysis decisions, many of which would not likely be reported in a publication. No two replicators reported the same sample size. Statistical significance varied across replications, and for one of the studies the effect's sign varied as well. The standard deviation of estimates across replications was 3–4 times the mean reported standard error.

Original language	English (US)
Pages (from-to)	944-960
Number of pages	17
Journal	Economic Inquiry
Volume	59
Issue number	3
DOIs	https://doi.org/10.1111/ecin.12992
State	Published - Jul 2021

All Science Journal Classification (ASJC) codes

General Business, Management and Accounting
Economics and Econometrics

Access to Document

10.1111/ecin.12992

Cite this

@article{d74ff0bbbea745dd922b4c7ad411258e,

title = "The influence of hidden researcher decisions in applied microeconomics",

abstract = "Researchers make hundreds of decisions about data collection, preparation, and analysis in their research. We use a many-analysts approach to measure the extent and impact of these decisions. Two published causal empirical results are replicated by seven replicators each. We find large differences in data preparation and analysis decisions, many of which would not likely be reported in a publication. No two replicators reported the same sample size. Statistical significance varied across replications, and for one of the studies the effect's sign varied as well. The standard deviation of estimates across replications was 3–4 times the mean reported standard error.",

author = "Nick Huntington-Klein and Andreu Arenas and Emily Beam and Marco Bertoni and Bloem, \{Jeffrey R.\} and Pralhad Burli and Naibin Chen and Paul Grieco and Godwin Ekpe and Todd Pugatch and Martin Saavedra and Yaniv Stopnitzky",

note = "Publisher Copyright: {\textcopyright} 2021 Western Economic Association International",

year = "2021",

month = jul,

doi = "10.1111/ecin.12992",

language = "English (US)",

volume = "59",

pages = "944--960",

journal = "Economic Inquiry",

issn = "0095-2583",

publisher = "Wiley-Blackwell Publishing Ltd",

number = "3",

}

TY - JOUR

T1 - The influence of hidden researcher decisions in applied microeconomics

AU - Huntington-Klein, Nick

AU - Arenas, Andreu

AU - Beam, Emily

AU - Bertoni, Marco

AU - Bloem, Jeffrey R.

AU - Burli, Pralhad

AU - Chen, Naibin

AU - Grieco, Paul

AU - Ekpe, Godwin

AU - Pugatch, Todd

AU - Saavedra, Martin

AU - Stopnitzky, Yaniv

PY - 2021/7

Y1 - 2021/7

N2 - Researchers make hundreds of decisions about data collection, preparation, and analysis in their research. We use a many-analysts approach to measure the extent and impact of these decisions. Two published causal empirical results are replicated by seven replicators each. We find large differences in data preparation and analysis decisions, many of which would not likely be reported in a publication. No two replicators reported the same sample size. Statistical significance varied across replications, and for one of the studies the effect's sign varied as well. The standard deviation of estimates across replications was 3–4 times the mean reported standard error.

AB - Researchers make hundreds of decisions about data collection, preparation, and analysis in their research. We use a many-analysts approach to measure the extent and impact of these decisions. Two published causal empirical results are replicated by seven replicators each. We find large differences in data preparation and analysis decisions, many of which would not likely be reported in a publication. No two replicators reported the same sample size. Statistical significance varied across replications, and for one of the studies the effect's sign varied as well. The standard deviation of estimates across replications was 3–4 times the mean reported standard error.

UR - https://www.scopus.com/pages/publications/85102799363

UR - https://www.scopus.com/inward/citedby.url?scp=85102799363&partnerID=8YFLogxK

U2 - 10.1111/ecin.12992

DO - 10.1111/ecin.12992

M3 - Article

AN - SCOPUS:85102799363

SN - 0095-2583

VL - 59

SP - 944

EP - 960

JO - Economic Inquiry

JF - Economic Inquiry

IS - 3

ER -

The influence of hidden researcher decisions in applied microeconomics

Abstract

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this