An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science

Obtaining a meaningful estimate of reproducibility requires conducting replications of a sizable number of studies. However, because of existing incentive structures, it is not in an individual scientist’s professional interest to conduct numerous replications. The Reproducibility Project addresses these barriers by spreading the workload over a large number of researchers. As of August 23, 2012, 72 volunteers from 41 institutions had joined the replication effort. Each contributor plays an important but circumscribed role, such as by contributing on a team conducting one replication study. Researchers volunteer to contribute on the basis of their interests, skills, and available resources. Information about the project’s coordination, planning, materials, and execution is available publicly on the Open Science Framework’s Web site (http://openscienceframework.org/). Open practices increase the accountability of the replication team and, ideally, the quality of the designs and results.

Studies eligible for replication were selected from 2008 issues of three prominent journals that differ in topical emphasis and publishing format (i.e., short reports vs. long-form articles): Journal of Experimental Psychology: Learning, Memory, and Cognition, Journal of Personality and Social Psychology, and Psychological Science.⁴ To minimize selection biases even within this restricted sample, replication teams choose from among the first 30 articles published in an issue. From the selected article, each team selects a key finding from a single study for replication (the last study by default, unless it is unfeasible to replicate). As eligible articles are claimed, additional articles from the sampling frame are made available for selection. Not all studies can be replicated. For example, some used unique samples or specialized equipment that is unavailable, and others were dependent on a specific historical event. Although feasibility constraints can reduce the generalizability of the ultimate results, they are inevitably part and parcel of reproducibility itself.

The project’s replication attempts follow a standardized protocol aimed at minimizing irrelevant variation in data collection and reporting methods, and maximizing the quality of replication efforts. The project attempts direct replications—“repetition of an experimental procedure” in order to “verify a piece of knowledge” (Schmidt, 2009, p. 92, 93). Replications must have high statistical power (1−β ≥ .80 for the effect size of the original study) and use the original materials, if they are available. Researchers solicit feedback on their research design from the original authors before collecting data, particularly to identify factors that may interfere with replication. Identified threats are either remedied with revisions or coded as potential predictors of reproducibility and written into the replication report.

Successful replication can be defined by “vote-counting,” either narrowly (i.e., obtaining the same statistically significant effect as original study) or broadly (i.e., obtaining a directionally similar, but not necessarily statistically significant, result), or quantitatively defined—for example, through meta-analytic estimates combining the original and replication study, comparisons of effect sizes, or updated estimates of Bayesian priors. As yet, there is no single general, standard answer to the question “What is replication?” so we employ multiple criteria (Valentine et al., 2011).

Failures to replicate might result from several factors. The first is a simple Type II error with an occurrence rate of 1−β: Some true findings will fail to replicate purely by chance. However, the overall replication rate can be measured against the average statistical power across studies. For this reason, the project focuses on the overall reproducibility rate. Individual studies that fail to replicate are not treated as disconfirmed. Failures to replicate can also occur if (a) the original effect is false; (b) the actual size of the effect is lower than originally reported, making it more difficult to detect; (c) the design, implementation, or analysis of either the original or replication study is flawed; or (d) the replication methodology differs from the original methodology in ways that are critical for successful replication.⁵ All of these reasons are important to consider in evaluations of reproducibility, but the most interesting may be the last. Identifying specific ways in which replications and original studies differ, especially when replications fail, can advance the theoretical understanding of previously unconsidered conditions necessary to obtain an effect. Thus, replication is theoretically consequential.

The most important point is that a failure to replicate an effect does not conclusively indicate that the original effect was false. An effect may also fail to replicate because of insufficient power, problems with the design of the replication study, or limiting conditions, whether known or unknown. For this reason, the Reproducibility Project investigates factors such as replication power, the evaluation of the replication-study design by the original authors, and the original study’s sample and effect sizes as predictors of reproducibility. Identifying the contribution of these factors to reproducibility is useful because each has distinct implications for interventions to improve reproducibility.

An estimate of the reproducibility of current psychological science will be an important first. A high reproducibility estimate might boost confidence in conventional research and peer-review practices in the face of criticisms about inappropriate flexibility in design, analysis, and reporting that can inflate the rate of false positives (Greenwald, 1975; John, Loewenstein, & Prelec, 2012; Simmons, Nelson, & Simonsohn, 2011). A low estimate might prompt reflection on the quality of standard practice, motivate further investigation of reproducibility, and ultimately lead to changes in practice and publishing standards (Bertamini & Munafò, 2012; LeBel & Peters, 2011).

Some may worry that the discovery of a low reproducibility rate will damage the image of psychology or of science more generally. It is certainly possible that opponents of science will use such a result to renew their calls to reduce funding for basic research. However, we believe that the alternative is much worse: having a low reproducibility rate, but failing to investigate and discover it. If reproducibility is lower than acceptable, then it is vitally important that we know about it in order to address it. Self-critique, and the promise of self-correction, is what makes science such an important part of humanity’s effort to understand nature and ourselves.

The Reproducibility Project uses an open methodology to test the reproducibility of psychological science. It also models procedures designed to simplify and improve reproducibility. Readers can review the discussion history of the project, examine the project’s design and structured protocol, retrieve replication materials from the various teams, obtain reports or raw data from completed replications, and join the project to conduct a replication (start here: http://openscienceframework.org/project/EZcUj/). Increasing the community of volunteers will strengthen the power and impact of the project. With this open, large-scale, collaborative scientific effort, we hope to identify the factors that contribute to the reproducibility and validity of psychological science. Ultimately, such evidence—and steps toward resolution, if the evidence produces a call for action—can improve psychological science’s most important asset: confidence in its methodology and findings.