Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results

Landy JF, Liam M, Ding IL, Viganola D, Tierney W, Dreber A, Johannesson M, Pfeiffer T, Ebersole CR, Gronau QF, Ly A, Bergh DVD, Marsman M, Derks K, Wagenmaker EJ, Proctor A, Bartels DM, Bauman CW, Brady WJ, Cheung F, Cimpian A, Dohle S, Donnellan MB, Hahn A, Hall MP, Jiménez-Leal W, Johnson DJ, Lucas RE, Monin B, Montealegre A, Mullen E, Pang J, Ray J, Reinero DA, Reynolds J, Sowden W, Storage D, Su R, Tworek CM, Van Bavel JJ, Walco D, Wills J, Xu X, Yam KC, Yang X, Cunningham WA, Schweinsberg M, Urwitz M, Uhlmann EL, Adamkovic M, Alaei R, Albers CJ, Allard A, Anderson IA, Andreychik MR, Babinčák P, Baker BJ, Baník G, Baskin E, Bavolar J, Berkers RM, Białek M, Blanke J, Breuer J, Brizi A, Brown SE, Brühlmann F, Bruns H, Caldwell L, Campourcy JF, Chan EY, Chang YP, Cheung BY, Chin A, Cho KW, Columbus S, Conway P, Corretti CA, Craig AW, Curran PG, Danvers AF, Dawson IG, Day MV, Dietl E, Doerflinger JT, Dominici A, Dranseika V, Edelsbrunner PA, Edlund JE, Fisher M, Fung A, Genschow O, Gnambs T, Goldberg MH, Graf-Vlachy L, Hafenbrack AC, Hafenbrädl S, Hartanto A, Heffner JP, Hilgard J, Holzmeister F, Horchak OV, Huang TS, Hüffmeier J, Hughes S, Hussey I, Imhoff R, Jaeger B, Jamro K, Johnson SG, Jones A, Keller L, Kombeiz O, Krueger LE, Lantian A, Laplante JP, Lazarevic LB, Leclerc J, Legate N, Leonhardt JM, Leung DW, Levitan CA, Lin H, Liu Q, Liuzza MT, Locke KD, Ly AL, MacEacheron M, Madan CR, Manley H, Mari S, Martončik M, McLean SL, McPhetres J, Mercier BG, Michels C, Mullarkey MC, Musser ED, Nalborczyk L, Nilsonne G, Otis NG, Otner SM, Otto PE, Oviedo-Trespalacios O, Paruzel-Czachura M, Pellegrini F, Pereira VM, Perfecto H, Pfuhl G, Phillips MH, Plonsky O, Pozzi M, Puric DB, Raymond-Barker B, Redman DE, Reynolds CJ, Ropovik I, Röseler L, Ruessmann JK, Ryan WH, Sablaturova N, Schuepfer KJ, Schütz A, Sirota M, Stefan M, Stocks EL, Strosser GL, Suchow JW, Szabelska A, Tey KS, Tiokhin L, Troian J, Utesch T, Vásquez-Echeverriá A, Vaughn LA, Verschoor M, Helversen BV, Wallisch P, Weissgerber SC, Wichman AL, Woike JK, Žeželj I, Zickfeld JH, Ahn Y, Blaettchen PF, Kang X, Lee YJ, Parker PM, Parker PA, Song JS, Very MA, Wong L (2020)

Publication Type: Journal article

Publication year: 2020

Journal

Book Volume: 146

Pages Range: 451-479

Journal Issue: 5

DOI: 10.1037/bul0000220

Abstract

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N = 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = —0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

Involved external institutions

University of Amsterdam Netherlands (NL) Universidad de los Andes - Colombia Colombia (CO) HarrisX / Harris Telecom LLC United States (USA) (US) Cardiovascular Research Foundation United States (USA) (US) Nova Southeastern University United States (USA) (US) University of Warwick United Kingdom (GB) National University of Singapore (NUS) Singapore (SG) George Mason University United States (USA) (US) University of Limerick Ireland (IE) Stockholm School of Economics (SSE) / Handelshögskolan i Stockholm (HHS) Sweden (SE) Massey University New Zealand (NZ) University of Virginia (UVA) United States (USA) (US) Nyenrode Business University Netherlands (NL) University of Chicago United States (USA) (US) University of California Irvine United States (USA) (US) The University of Hong Kong (HKU) / 香港大學 Hong Kong (HK) Universität zu Köln Germany (DE) Michigan State University United States (USA) (US) University of Michigan United States (USA) (US) University of Maryland, College Park (UMD) United States (USA) (US) Stanford University United States (USA) (US) San José State University United States (USA) (US) Renmin University of China (RUC) / 中国人民大学 China (CN) Abilene Christian University United States (USA) (US) University of Denver United States (USA) (US) Duke University United States (USA) (US) Hainan Medical University (HMU) / 海南医学院 China (CN) Catholic University of the Sacred Heart / Università Cattolica del Sacro Cuore Italy (IT) Tsinghua University China (CN) University of Toronto Canada (CA) ESMT European School of Management and Technology Germany (DE) Insead - The Business School for the World France (FR) University of Prešov / Prešovská univerzita v Prešove Slovakia (SK) Université Paris VIII France (FR) Texas A&M University United States (USA) (US) University of Texas at Dallas (UTD / UT Dallas) United States (USA) (US) University of Kentucky United States (USA) (US) Grand Valley State University (GVSU) United States (USA) (US) University of Washington United States (USA) (US) Southern Utah Univ (SUU) United States (USA) (US) Johannes Kepler Universität (JKU) Linz Austria (AT) Yale University United States (USA) (US) Universität Passau Germany (DE) Stevens Institute of Technology (SIT) United States (USA) (US) Brown University United States (USA) (US) Illinois State University (ISU) United States (USA) (US) Leopold-Franzens-Universität Innsbruck / University of Innsbruck Austria (AT) Iscte – Instituto Universitário de Lisboa (ISCTE-IUL) / Instituto Superior de Ciências do Trabalho e da Empresa Portugal (PT) University College London (UCL) United Kingdom (GB) Technische Universität Dortmund Germany (DE) Universiteit Gent (UGent) / Ghent University Belgium (BE) Concordia University Canada (CA) Illinois Institute of Technology United States (USA) (US) Chulalongkorn University / จุฬาลงกรณ์มหาวิทยาลัย Thailand (TH) University of Milano-Bicocca / Università degli Studi di Milano-Bicocca, UNIMIB Italy (IT) Universität Münster (ehem. Westfälische Wilhelms-Universität (WWU)) Germany (DE) Walden University United States (USA) (US) University of Regina Canada (CA) Ithaca College United States (USA) (US) University of Texas at Austin United States (USA) (US) Florida International University United States (USA) (US) Karolinska Institute Sweden (SE) University of California, Berkeley (UC Berkeley) United States (USA) (US) Imperial College London / The Imperial College of Science, Technology and Medicine United Kingdom (GB) Europa-Universität Viadrina Frankfurt (Oder) Germany (DE) Queensland University of Technology (QUT) Australia (AU) University of Silesia in Katowice / Uniwersytet Śląski w Katowicach Poland (PL) University of Padua / Universita degli Studi di Padova Italy (IT) University of Lisbon / Universidade de Lisboa (ULisboa) Portugal (PT) Washington University in St. Louis United States (USA) (US) Universitetet i Tromsø / The Arctic University of Norway (UiT) Norway (NO) Otto-Friedrich-Universität Bamberg Germany (DE) University of Essex United Kingdom (GB) The University of Texas at Tyler (UTT) United States (USA) (US) Queen's University United Kingdom (GB) University of Groningen / Rijksuniversiteit Groningen Netherlands (NL) Fairfield University United States (USA) (US) University of Massachusetts Boston (UMass Boston) United States (USA) (US) Pavol Jozef Šafárik University in Kosice / Univerzita Pavla Jozefa Šafárika v Košiciach Slovakia (SK) Max-Planck-Institut für Kognitions- und Neurowissenschaften / Max Planck Institute for Human Cognitive and Brain Sciences Germany (DE) Akademia Leona Koźmińskiego / Kozminski University Poland (PL) GESIS - Leibniz-Institut für Sozialwissenschaften Germany (DE) Università degli studi "La Sapienza" Italy (IT) Universität Basel Switzerland (CH) Universität Hamburg (UHH) Germany (DE) Irrational Agency Ltd. United Kingdom (GB) University of Clermont Auvergne France (FR) Monash University Australia (AU) Academia Sinica / 中央研究院 Taiwan (TW) University of British Columbia Canada (CA) Public Company Accounting Oversight Board PCAOB United States (USA) (US) University of Houston United States (USA) (US) Vrije Universiteit Amsterdam (VU) / University Amsterdam Netherlands (NL) Florida State University (FSU) United States (USA) (US) University of Arizona United States (USA) (US) University of Southampton United Kingdom (GB) Memorial University of Newfoundland (MUN) Canada (CA) Loughborough University United Kingdom (GB) Universität Konstanz Germany (DE) European University Institute Italy (IT) Kaunas University of Technology (KTU) Lithuania (LT) Eidgenössische Technische Hochschule Zürich (ETHZ) / Swiss Federal Institute of Technology in Zurich Switzerland (CH) Rochester Institute of Technology (RIT) United States (USA) (US) Southern Methodist University (SMU) United States (USA) (US) Johannes Gutenberg-Universität Mainz (JGU) Germany (DE) Tilburg University Netherlands (NL) UMass University of Massachusetts Dartmouth United States (USA) (US) University of Bath United Kingdom (GB) The University of Liverpool United Kingdom (GB) Eindhoven University of Technology / Technische Universiteit Eindhoven (TU/e) Netherlands (NL) Texas A&M University, Commerce United States (USA) (US) University of Paris 10 - Nanterre / Université Paris Nanterre La Défense France (FR) Clark University United States (USA) (US) University of Belgrade / Универзитет у Београду Serbia (RS) University of Nevada, Reno United States (USA) (US) City University of New York (CUNY) United States (USA) (US) Occidental College United States (USA) (US) Istanbul Bilgi University / İstanbul Bilgi Üniversitesi Turkey (TR) Hubei University / 湖北大学 China (CN) Magna Græcia University of Catanzaro / Università degli studi Magna Græcia di Catanzaro Italy (IT) University of Idaho United States (USA) (US) Loma Linda University (LLU) United States (USA) (US) Western University Canada (CA) University of Nottingham United Kingdom (GB) Max-Planck-Institut für Bildungsforschung (MPIB) / Max Planck Institute for Human Development Germany (DE) University of Roehampton United Kingdom (GB) Pacific Lutheran University United States (USA) (US) Universität Bremen Germany (DE) University of Oslo Norway (NO) Western Kentucky University United States (USA) (US) Masaryk University Czech Republic (CZ) University of Miami United States (USA) (US) Universität Kassel Germany (DE)

How to cite

APA:

Landy, J.F., Liam, M., Ding, I.L., Viganola, D., Tierney, W., Dreber, A.,... Wong, L. (2020). Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results. Psychological Bulletin, 146(5), 451-479. https://doi.org/10.1037/bul0000220

MLA:

Landy, Justin F., et al. "Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results." Psychological Bulletin 146.5 (2020): 451-479.

BibTeX: Download