![[PDF]](/pb-assets/Icons/adobe-pdf-icon.png){kind=icon}
Abstract
In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.
References
| Babcock, L., Loewenstein, G. (1997). Explaining bargaining impasse: The role of self-serving biases. Journal of Economic Perspectives, 11, 109–126. Google Scholar, Crossref, ISI | |
| Dawson, E., Gilovich, T., Regan, D. T. (2002). Motivated reasoning and performance on the Wason Selection Task. Personality and Social Psychology Bulletin, 28, 1379–1387. Google Scholar, SAGE Journals, ISI | |
| Gilovich, T. (1983). Biased evaluation and persistence in gambling. Journal of Personality and Social Psychology, 44, 1110–1126. Google Scholar, Crossref, Medline, ISI | |
| Hastorf, A. H., Cantril, H. (1954). They saw a game: A case study. Journal of Abnormal and Social Psychology, 49, 129– 134. Google Scholar, Crossref, Medline, ISI | |
| Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. Retrieved from http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 Google Scholar | |
| John, L., Loewenstein, G. F., Prelec, D. (2011). Measuring the prevalence of questionable research practices with incentives for truth-telling. Manuscript submitted for publication. Google Scholar | |
| Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480–498. Google Scholar, Crossref, Medline, ISI | |
| Pocock, S. J. (1977). Group sequential methods in the design and analysis of clinical trials. Biometrika, 64, 191–199. Google Scholar, Crossref, ISI | |
| Schooler, J. (2011). Unpublished results hide the decline effect. Nature, 470, 437. Google Scholar | |
| Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. (2011). Why psychologists must change the way they analyze their data: The case of psi. Journal of Personality and Social Psychology, 100, 426–432. Google Scholar, Crossref, Medline, ISI | |
| Zuckerman, M. (1979). Attribution of success and failure revisited, or: The motivational bias is alive and well in attribution theory. Journal of Personality, 47, 245–287. Google Scholar, Crossref, ISI |
