Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017)

Psychologists are navigating an unprecedented period of introspection about the credibility and utility of their discipline. Reform initiatives emphasize the benefits of transparency and reproducibility-related research practices; however, adoption across the psychology literature is unknown. Estimating the prevalence of such practices will help to gauge the collective impact of reform initiatives, track progress over time, and calibrate future efforts. To this end, we manually examined a random sample of 250 psychology articles published between 2014 and 2017. Over half of the articles were publicly available (154/237, 65%, 95% confidence interval [CI] = [59%, 71%]); however, sharing of research materials (26/183; 14%, 95% CI = [10%, 19%]), study protocols (0/188; 0%, 95% CI = [0%, 1%]), raw data (4/188; 2%, 95% CI = [1%, 4%]), and analysis scripts (1/188; 1%, 95% CI = [0%, 1%]) was rare. Preregistration was also uncommon (5/188; 3%, 95% CI = [1%, 5%]). Many articles included a funding disclosure statement (142/228; 62%, 95% CI = [56%, 69%]), but conflict-of-interest statements were less common (88/228; 39%, 95% CI = [32%, 45%]). Replication studies were rare (10/188; 5%, 95% CI = [3%, 8%]), and few studies were included in systematic reviews (21/183; 11%, 95% CI = [8%, 16%]) or meta-analyses (12/183; 7%, 95% CI = [4%, 10%]). Overall, the results suggest that transparency and reproducibility-related research practices were far from routine. These findings establish baseline prevalence estimates against which future progress toward increasing the credibility and utility of psychology research can be compared.

(1) For the questions "How does the statement indicate the materials are available?" and "How does the statement indicate the data are available?" we added an explicit response option: "Appendix within the present article".
(2) For the questions "Does the article state whether or not the materials are available?" and "Does the article state whether or not the data are available?" we added an explicit response option "Source of data/materials provided but no explicit availability statement". However, because this was introduced late in the data collection process it was not applied consistently and we have elected to re-code this response option as "No -there is no data availability statement". (3) For the questions "What type of study is being reported?" we changed the response option 'survey' to 'survey/interview' and renamed 'field study' to 'observational study'. We did this to better match our coding decisions. (4) For the citation evaluation, we added two questions: "How many times was the article cited incidentally in a meta-analysis (i.e., no intention to include in data synthesis)?" and "How many times was the article cited incidentally in a systematic review (i.e., no intention to include in formal review)?" (5) For the questions "Does the article include a statement indicating whether there were funding sources?" we added the response option "Yes -the statement identifies funding sources but the private/public status is unclear" (6) To concisely present the content of conflict of interest statements we categorized them into the types shown in Table 3. This analysis was not pre-registered. (7) To align the extraction form with previous studies (e.g., Wallach et al., 2019) we included an additional response option ("Empirical datacost effectiveness and/or decision analysis") for the question "What type of study is being reported?"
All coding differences were resolved through discussion.
For all variables there was substantial agreement, however, for the four variables with the lowest Fleiss' kappa (<.70), we conducted an additional inspection of the differences to explore if there were any issues of concern. In summary, few differences arose from substantive disagreement between coders and were more often due to earlier differences coding study type (which could render later variables irrelevant) or differences in how non-standard 'other' responses were recorded in the extraction form. Details about the differences are provided below.

Coding differences for data availability statements
For the variable "Does the article state whether or not data are available?", there were 13 cases of coder disagreement. In two cases (wfFwp, XIWav), one coder appeared to have missed a relevant data availability statement. In two cases (bppGq, VFLFB) one coder did not search for a data availability statement because there was an earlier disagreement about how to code the study type. In 5 cases (hkNiS, KuEds, ldnNm, posTo, snvlH), one coder had used an explicit response option -"Source of data provided but no explicit availability statement" -that was introduced late in the data collection process but later re-coded to "No -there is no data availability statement" to ensure consistency (see Supplementary Information A, point 2). In one case (LaZfV) the first coder identified availability of a resource that was later determined to be materials rather than data. In one case (tIUAU), the first coder identified a statement regarding the availability of preliminary data but it was later determined that this did not pertain to the main data underlying the reported study so would be more accurately classified as "No -there is no data availability statement". In one case (VEDlR), the first coder identified a statement which said 'supplementary data' were available, but further inspection suggested that these were not data so this was not considered a data availability statement. In one case (zFYIM), both coders used the 'other' response to record that data access required a fee, but as this was a non-standardized response option, the text differed.

Coding differences for pre-registration availability statements
For the variable "Does the article state whether or not the study (or some aspect of the study) was pre-registered?", there were 5 cases of coder disagreement. In two cases (HdNmC, DfkgD), one coder appeared to have missed a relevant pre-registration availability statement. In one case (BNDMz), one coder identified a clinical trial registration number, but after discussion the article was determined to be a clinical trial protocol, rather than a completed study, and the study type was thus re-classified as "No empirical data", rendering the pre-registration variable irrelevant. In two cases (bppGq, VFLFB) one coder did not search for a pre-registration availability statement because there was an earlier disagreement about how to code the study type.

Coding differences for meta-analysis exclusions
For the variable "How many times has the article been explicitly excluded from a metaanalysis?", there were 5 cases of coder disagreement. In two cases (pxYoP, QdhZZ), one coder missed relevant information. In one case (CywSU) a coder incorrectly recorded that there was an exclusion; it was later agreed that there were none. In two cases (lvyGE, snvlH), one coder did not search for meta-analysis exclusions because there was an earlier disagreement about how to code the study type.

Coding differences for cited by a replication
For the variable "How many times has the article been cited in a replication study?", there were 6 cases of coder disagreement. In two cases (pxYoP, QdhZZ), one coder missed relevant information. In two cases (gyYOp, xOYcP), the status of a citing article as a replication was ambiguous, but following discussion the coders agreed that because the authors of those articles had not explicitly described their studies as replications, we would not classify them as such. In two cases (lvyGE, snvlH), one coder did not search for meta-analysis exclusions because there was an earlier disagreement about how to code the study type.