Questionable research practices may have little effect on replicability

This article examines why many studies fail to replicate statistically significant published results. We address this issue within a general statistical framework that also allows us to include various questionable research practices (QRPs) that are thought to reduce replicability. The analyses indi...

Full description

Bibliographic Details
Main Authors: Rolf Ulrich, Jeff Miller
Format: Article
Language:English
Published: eLife Sciences Publications Ltd 2020-09-01
Series:eLife
Subjects:
Online Access:https://elifesciences.org/articles/58237
id doaj-4250a04349624728a66e927ad62a0aad
record_format Article
spelling doaj-4250a04349624728a66e927ad62a0aad2021-05-05T21:31:05ZengeLife Sciences Publications LtdeLife2050-084X2020-09-01910.7554/eLife.58237Questionable research practices may have little effect on replicabilityRolf Ulrich0https://orcid.org/0000-0001-8443-2705Jeff Miller1https://orcid.org/0000-0003-2718-3153Department of Psychology, University of Tübingen, Tübingen, GermanyDepartment of Psychology, University of Otago, Dunedin, New ZealandThis article examines why many studies fail to replicate statistically significant published results. We address this issue within a general statistical framework that also allows us to include various questionable research practices (QRPs) that are thought to reduce replicability. The analyses indicate that the base rate of true effects is the major factor that determines the replication rate of scientific results. Specifically, for purely statistical reasons, replicability is low in research domains where true effects are rare (e.g., search for effective drugs in pharmacology). This point is under-appreciated in current scientific and media discussions of replicability, which often attribute poor replicability mainly to QRPs.https://elifesciences.org/articles/58237meta-researchp-hackingbase rate of true effectsreplicabilityfalse positivesmathematical modelling of research process
collection DOAJ
language English
format Article
sources DOAJ
author Rolf Ulrich
Jeff Miller
spellingShingle Rolf Ulrich
Jeff Miller
Questionable research practices may have little effect on replicability
eLife
meta-research
p-hacking
base rate of true effects
replicability
false positives
mathematical modelling of research process
author_facet Rolf Ulrich
Jeff Miller
author_sort Rolf Ulrich
title Questionable research practices may have little effect on replicability
title_short Questionable research practices may have little effect on replicability
title_full Questionable research practices may have little effect on replicability
title_fullStr Questionable research practices may have little effect on replicability
title_full_unstemmed Questionable research practices may have little effect on replicability
title_sort questionable research practices may have little effect on replicability
publisher eLife Sciences Publications Ltd
series eLife
issn 2050-084X
publishDate 2020-09-01
description This article examines why many studies fail to replicate statistically significant published results. We address this issue within a general statistical framework that also allows us to include various questionable research practices (QRPs) that are thought to reduce replicability. The analyses indicate that the base rate of true effects is the major factor that determines the replication rate of scientific results. Specifically, for purely statistical reasons, replicability is low in research domains where true effects are rare (e.g., search for effective drugs in pharmacology). This point is under-appreciated in current scientific and media discussions of replicability, which often attribute poor replicability mainly to QRPs.
topic meta-research
p-hacking
base rate of true effects
replicability
false positives
mathematical modelling of research process
url https://elifesciences.org/articles/58237
work_keys_str_mv AT rolfulrich questionableresearchpracticesmayhavelittleeffectonreplicability
AT jeffmiller questionableresearchpracticesmayhavelittleeffectonreplicability
_version_ 1721457975429693440