![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
The issue of a published literature not representative of the population of research is most often discussed in terms of entire studies being suppressed. However, alternative sources of publication bias are questionable research practices (QRPs) that entail post hoc alterations of hypotheses to support data or post hoc alterations of data to support hypotheses. Using general strain theory as an explanatory framework, we outline the means, motives, and opportunities for researchers to better their chances of publication independent of rigor and relevance. We then assess the frequency of QRPs in management research by tracking differences between dissertations and their resulting journal publications. Our primary finding is that from dissertation to journal article, the ratio of supported to unsupported hypotheses more than doubled (0.82 to 1.00 versus 1.94 to 1.00). The rise in predictive accuracy resulted from the dropping of statistically nonsignificant hypotheses, the addition of statistically significant hypotheses, the reversing of predicted direction of hypotheses, and alterations to data. We conclude with recommendations to help mitigate the problem of an unrepresentative literature that we label the “Chrysalis Effect.”
References
|
Agnew, R. 1992. Foundation for a general strain theory of crime and delinquency. Criminology, 30: 47-88. Google Scholar | Crossref | ISI | |
|
Agnew, R., Brezina, T., Wright, J. P., Cullen, F. T. 2002. Strain, personality traits, and delinquency: Extending general strain theory. Criminology, 40: 43-72. Google Scholar | Crossref | ISI | |
|
Aguinis, H., Gottfredson, R. K., Joo, H. in press. Best-practice recommendations for defining, identifying, and handling outliers. Organizational Research Methods. Google Scholar | |
|
Academy of Management . 2006. Academy of Management code of ethics. Retrieved March 1, 2013, from http://aom.org/uploadedFiles/About_AOM/Governance/AOM_Code_of_Ethics.pdf Google Scholar | |
|
American Psychological Association . 2010. American Psychological Association ethical principles of psychologists and code of conduct. Retrieved March 1, 2013, from http://www.apa.org/ethics/code2002.html Google Scholar | |
|
Ariely, D. 2012. The (honest) truth about dishonesty: How we lie to everyone-especially ourselves. New York, NY: HarperCollins. Google Scholar | |
|
Bakker, M., van Dijk, A., Wicherts, J. M. 2012. The rules of the game called psychological science. Perspectives on Psychological Science, 7: 543-554. Google Scholar | SAGE Journals | ISI | |
|
Banks, G. C., McDaniel, M. A. 2011. The kryptonite of evidence-based I-O psychology. Industrial and Organizational Psychology, 4: 40-44. Google Scholar | Crossref | ISI | |
|
Banks, G. C., O’Boyle, E. H. 2013. Why we need I-O psychology to fix I-O psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6: 284-287. Google Scholar | Crossref | ISI | |
|
Bedeian, A. G. 2003. The manuscript review process: The proper roles of authors, referees, and editors. Journal of Management Inquiry, 12: 331-338. Google Scholar | SAGE Journals | ISI | |
|
Bedeian, A. G. 2004. Peer review and the social construction of knowledge in the management discipline. Academy of Management Learning & Education, 3: 198-216. Google Scholar | Crossref | ISI | |
|
Bedeian, A. G., Taylor, S. G., Miller, A. N. 2010. Management science on the credibility bubble: Cardinal sins and various misdemeanors. Academy of Management Learning & Education, 9: 715-725. Google Scholar | Crossref | ISI | |
|
Bracken, M. B., Sinclair, J. C. (1998). When can odds ratios mislead? Avoidable systematic error in estimating treatment effects must not be tolerated. British Medical Journal, 317:1156-1157. Google Scholar | Medline | |
|
Briner, R. B., Rousseau, D. M. 2011. Evidence-based I-O psychology: Not there yet. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4: 3-22. Google Scholar | Crossref | ISI | |
|
Buchheit, S., Collins, D., Reitenga, A. 2002. A cross-discipline comparison of top-tier academic journal publication rates: 1997-1999. Journal of Accounting Education, 20: 123-130. Google Scholar | Crossref | |
|
Cashen, L. H., Geiger, S. W. 2004. Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies. Organizational Research Methods, 7: 151-167. Google Scholar | SAGE Journals | ISI | |
|
Chalmers, I. 1990. Underreporting research is scientific misconduct. Journal of the American Medical Association, 263: 1405-1408. Google Scholar | Crossref | Medline | ISI | |
|
Chan, A. W., Hrobjartsson, A., Jorgensen, K. J., Gotzsche, P. C., Altman, D. G. 2008. Discrepancies in sample size calculations and data analyses reported in randomised trials: Comparison of publications with protocols. British Medical Journal, 337, 1-8. Google Scholar | Crossref | |
|
Chen, X. P. 2011. Author ethical dilemmas in the research publication process. Management and Organization Review, 7: 423-432. Google Scholar | Crossref | ISI | |
|
Cloward, R. A., Ohlin, L. E. 1960. Delinquency and opportunity. New York, NY: Free Press. Google Scholar | |
|
Cohen, A. K. 1955. Delinquent boys. New York, NY: Free Press. Google Scholar | |
|
Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20: 37-46. Google Scholar | SAGE Journals | ISI | |
|
Cohen, J. 1990. Things I have learned (thus far). American Psychologist, 45: 1304-1312. Google Scholar | Crossref | ISI | |
|
Cohen, J. 1994. The earth is round (p < .05). American Psychologist, 49: 997-1003. Google Scholar | Crossref | ISI | |
|
Corley, K. G., Gioia, D. A. 2000. The rankings game: Managing business school reputation. Corporate Reputation Review, 3: 319-333. Google Scholar | Crossref | |
|
Crook, T. R., Shook, C. L., Morris, M. L., Madden, T. M. 2010. Are we there yet? An assessment of research design and construct measurement practices in entrepreneurship research. Organizational Research Methods, 13: 192-206. Google Scholar | SAGE Journals | ISI | |
|
Dalton, D. R., Aguinis, H., Dalton, C. M., Bosco, F. A., Pierce, C. A. 2012. Revisiting the file drawer problem in meta-analysis: An assessment of published and nonpublished correlation matrices. Personnel Psychology, 65: 221-249. Google Scholar | Crossref | ISI | |
|
Davies, H. T. O., Crombie, I. K., Tavakoli, M. (1998). When can odds ratios mislead? British Medical Journal, 316: 989-991. Google Scholar | Crossref | Medline | |
|
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A. W., Cronin, E., Williamson, P. R. 2008. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One, 3: e3081. Google Scholar | Crossref | Medline | ISI | |
|
Dwan, K., Altman, D. G., Cresswell, L., Blundell, M., Gamble, C., Williams, P. R. 2011. Comparison of protocols and registry entries to published reports for randomised controlled trials. The Cochrane Collaboration. Google Scholar | |
|
Eden, D. 2002. From the editors: Replication, meta-analysis, scientific progress, and AMJ’s publication policy. Academy of Management Journal, 45: 841-846. Google Scholar | Crossref | ISI | |
|
Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J., Brand, R. A., Leopold, S. S. 2008. Testing for the presence of positive-outcome bias in peer review: A randomized controlled trial. Archives of Internal Medicine, 170: 1934-1939. Google Scholar | |
|
Fanelli, D. 2010a. Do pressures to publish increase scientists’ bias? An empirical support from United States data. PLoS One, 5: e10271. Google Scholar | Crossref | Medline | ISI | |
|
Fanelli, D. 2010b. “Positive” results increase down the hierarchy of the sciences. PLoS One, 5: e10068. Google Scholar | Crossref | Medline | ISI | |
|
Fanelli, D. 2012. Negative results are disappearing from most disciplines and countries. Scientometrics, 90: 891-904. Google Scholar | Crossref | ISI | |
|
Fischer, C. T. 1970. Levels of cheating under conditions of informative appeal to honesty, public affirmation of value, and threats of punishment. Journal of Educational Research, 64: 12-16. Google Scholar | Crossref | ISI | |
|
Frank, M. C., Saxe, R. 2012. Teaching replication. Perspectives on Psychological Science, 7: 600-604. Google Scholar | SAGE Journals | ISI | |
|
Gerber, A. S., Malhotra, N. 2008. Publication bias in empirical sociological research: Do arbitrary significance levels distort published results? Sociological Methods and Research, 37: 3-30. Google Scholar | SAGE Journals | ISI | |
|
Gigerenzer, G. 2004. Mindless statistics. Journal of Socio-Economics, 33: 587-606. Google Scholar | Crossref | |
|
Gioia, D., Corley, K. G. 2002. Being good versus looking good: Business school rankings and the Circean transformation from substance to image. Academy of Management Learning and Education, 1: 107-120. Google Scholar | Crossref | ISI | |
|
Gomez-Mejia, L. R., Balkin, D. B. 1992. Determinants of faculty pay: An agency theory perspective. Academy of Management Journal, 35: 921-955. Google Scholar | Crossref | ISI | |
|
Guttmacher, A. E., Nabel, E. G., Collins, F. S. 2009. Why data-sharing policies matter. Proceedings of the National Academy of Sciences, 106: 16894. Google Scholar | Crossref | Medline | ISI | |
|
Hahn, S., Williamson, P. R., Hutton, J. L., Garner, P., Flynn, E. V. 2000. Assessing the potential for bias in meta-analysis due to selective reporting of subgroup analyses within studies. Statistics in Medicine, 19: 3325-3336. Google Scholar | Crossref | Medline | ISI | |
|
Hambrick, D. C. 2007. The field of management’s devotion to theory: Too much of a good thing? Academy of Management Journal, 50: 1346-1352. Google Scholar | Crossref | ISI | |
|
Hardin, G. 1968. The tragedy of the commons. Science, 162, 1243-1248. Google Scholar | Crossref | Medline | ISI | |
|
Harter, J. K., Schmidt, F. L. 2008. Conceptual versus empirical distinctions among constructs: Implications for discriminant validity. Industrial and Organizational Psychology, 1: 36-39. Google Scholar | Crossref | ISI | |
|
Harzing, A. W., van der Wal, R. 2008. A Google Scholar h-index for journals: An alternative metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60: 41-46. Google Scholar | Crossref | |
|
Hirsch, J. E. 2005. An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102: 16569-16572. Google Scholar | Crossref | Medline | ISI | |
|
Huic, M., Marusic, M., Marusic, A. 2011. Completeness and changes in registered data and reporting bias of randomized controlled trials in ICMJE journals after trial registration policy. PLoS One, 6: 9. Google Scholar | Crossref | ISI | |
|
Ioannidis, J. P. A. 2005. Why most published research findings are false. PLoS Medicine, 2: e124. Google Scholar | Crossref | Medline | ISI | |
|
Ioannidis, J. P. A. 2008. Why most discovered true associations are inflated. Epidemiology, 19: 640-648. Google Scholar | Crossref | Medline | ISI | |
|
Jick, T. D. 1979. Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24: 602-611. Google Scholar | Crossref | ISI | |
|
John, L. K., Loewenstein, G., Prelec, D. 2012. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23: 524-532. Google Scholar | SAGE Journals | ISI | |
|
Kacmar, K. M. 2009. From the editors: An ethical quiz. Academy of Management Journal, 52: 432-434. Google Scholar | Crossref | ISI | |
|
Kemper, T. D. 1978. A social interactional theory of emotions. New York, NY: Wiley. Google Scholar | |
|
Kepes, S., McDaniel, M. A. 2013. How trustworthy is the scientific literature in I-O psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6: 252-268. Google Scholar | Crossref | ISI | |
|
Kerr, N. L. 1998. HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2: 196-217. Google Scholar | SAGE Journals | |
|
Kirkham, J. J., Riley, R. D., Williams, P. R. 2011. A multivariate meta-analysis approach for reducing the impact of outcome reporting bias in systematic reviews. Statistics in Medicine, 31: 2179-2195. Google Scholar | Crossref | ISI | |
|
Kish-Gephart, J. J., Harrison, D. A., Treviño, L. K. 2010. Bad apples, bad cases, and bad barrels: Meta-analytic evidence about sources of unethical decisions at work. Journal of Applied Psychology, 95: 1-31. Google Scholar | Crossref | Medline | ISI | |
|
Klein, O., Doyen, S., Leys, C., Miller, S., Questienne, L., Cleeremans, A. 2012. Low hopes, high expectations, expectancy effects, and the replicability of behavioral experiments. Perspectives on Psychological Science, 7: 572-584. Google Scholar | SAGE Journals | ISI | |
|
Knol, M. J., Duijnhoven, R. G., Grobbee, D. E., Moons, K. G., Groenwold, R. H. (2011). Potential misinterpretation of treatment effects due to use of odds ratios and logistic regression in randomized controlled trials. PLoS One, 6: e21248. Google Scholar | Crossref | Medline | ISI | |
|
Leavitt, K., Mitchell, T. R., Peterson, J. 2010. Theory pruning: Strategies to reduce our dense theoretical landscape. Organizational Research Methods, 13: 644-667. Google Scholar | SAGE Journals | ISI | |
|
Le, H., Oh, I.-S., Shaffer, J., Schmidt, F. L. 2007. Implications of methodological advances for the practice of personnel selection: How practitioners benefit from meta-analysis. Academy of Management Perspectives, 21: 6-15. Google Scholar | Crossref | ISI | |
|
Leung, K. 2011. Presenting post hoc hypotheses as a priori: Ethical and theoretical issues. Management and Organization Review, 7: 85-94. Google Scholar | Crossref | ISI | |
|
Locke, E. A., Latham, G. P. 1990. A theory of goal setting and task performance. Upper Saddle River, NJ: Prentice Hall. Google Scholar | |
|
MacDonald, S., Kam, J. 2007. Ring a ring o’ roses: Quality journals and gamesmanship in management studies. Journal of Management Studies, 44: 640-55. Google Scholar | Crossref | ISI | |
|
Marszalek, J. M., Barber, C., Kohlhart, J., Holmes, C. B. 2011. Sample size in psychological research over the past 30 years. Perceptual and Motor Skills, 112: 331-348. Google Scholar | SAGE Journals | ISI | |
|
Martinson, B. C., Anderson, M. S., De Vries, R. 2005. Scientists behaving badly. Nature, 435: 737-738. Google Scholar | Crossref | Medline | ISI | |
|
Masicampo, E. J., Lalande, D. R. 2012. A peculiar prevalence of p values just below. 05. Quarterly Journal of Experimental Psychology, 65: 2271-2279. Google Scholar | SAGE Journals | ISI | |
|
Mathieu, S., Boutron, I., Moher, D., Altman, D. G., Ravaud, P. 2009. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA: The Journal of the American Medical Association, 302: 977-984. Google Scholar | Crossref | Medline | ISI | |
|
Mazar, N., Amir, O., Ariely, D. 2008. The dishonesty of honest people: A theory of self-concept maintenance. Journal of Marketing Research, 45: 633-644. Google Scholar | SAGE Journals | ISI | |
|
McCabe, D. L., Trevino, L. K., Butterfield, K. D. 1996. The influence of collegiate and corporate codes of conduct on ethics-related behavior in the workplace. Business Ethics Quarterly, 6: 461-476. Google Scholar | Crossref | |
|
Meehl, P. E. 1978. Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46: 806-834. Google Scholar | Crossref | ISI | |
|
Merton, R. K. 1938. Social structure and anomie. American Sociological Review, 3: 672-682. Google Scholar | Crossref | |
|
Mills, J. L. 1993. Data torturing. New England Journal of Medicine, 329: 1196-1199. Google Scholar | Crossref | Medline | ISI | |
|
Nicholas, J. M., Katz, M. 1985. Research methods and reporting practices in organization development: A review and some guidelines. Academy of Management Review, 10: 737-749. Google Scholar | Crossref | ISI | |
|
Nickerson, R. S. 2000. Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5: 241-301. Google Scholar | Crossref | Medline | ISI | |
|
Orlitzky, M. 2012. How can significance tests be deinstitutionalized? Organizational Research Methods, 15: 199-228. Google Scholar | SAGE Journals | ISI | |
|
Popper, K. R. 1968. The logic of scientific discovery. New York, NY: Harper and Row. Google Scholar | |
|
Rosenthal, R. 1990. Replication in behavioral research. Journal of Social Behavior and Personality, 5: 1-30. Google Scholar | |
|
Rynes, S., Gerhart, B. 2003. Compensation: Theory, evidence, and strategic implications. Thousand Oaks, CA: Sage. Google Scholar | |
|
Sackett, D. L., Deeks, J. J., Altman, D. G. (1996). Down with odds ratios! Evidence Based Medicine, 1: 164-166. Google Scholar | |
|
Sackett, P. R., Larson, J. R. 1990. Research strategies and tactics in industrial and organizational psychology. In Dunnette, M. D., Hough, L. M. (Eds.), Handbook of industrial and organizational psychology, vol. 1: 419-489. Palo Alto, CA: Consulting Psychologists Press. Google Scholar | |
|
Schmidt, F., Hunter, J. 2002. Are there benefits from NHST? American Psychologist, 57: 65-66. Google Scholar | Crossref | Medline | ISI | |
|
Schmidt, F. L. 1992. What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47: 1173-1181. Google Scholar | Crossref | ISI | |
|
Schmidt, F. L. 1996. Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1: 115-129. Google Scholar | Crossref | ISI | |
|
Schminke, M. 2009. Editor’s comments: The better angels of our nature-ethics and integrity in the publishing process. Academy of Management Review, 34: 586-591. Google Scholar | Crossref | ISI | |
|
Siemens, J. C., Burton, S., Jensen, T., Mendoza, N. A. T. 2005. An examination of the relationship between research productivity in prestigious business journals and popular press. Journal of Business Research, 58: 467-76. Google Scholar | Crossref | ISI | |
|
Simmons, J. P., Nelson, L. D., Simonsohn, U. 2011. False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22: 1359-1366. Google Scholar | SAGE Journals | ISI | |
|
Stroebe, W., Postmes, T., Spears, R. 2012. Scientific misconduct and the myth of self-correction in science. Perspective of Psychological Science, 7: 670-688. Google Scholar | SAGE Journals | ISI | |
|
Tsang, E. W., Frey, B. S. 2007. The as-is journal review process: Let authors own their ideas. Academy of Management Learning and Education, 6: 128-136. Google Scholar | Crossref | ISI | |
|
Turner, E. H., Knoepflmacher, D., Shapley, L. 2012. Publication bias in antipsychotic trials: An analysis of efficacy comparing the published literature to the US Food and Drug Administration database. PLoS Medicine, 9: 1-17. Google Scholar | Crossref | ISI | |
|
Wagenmakers, E. J. 2007. A practical solution to the pervasive problems of p values. Psychonomic Bulletin and Review, 14: 779-804. Google Scholar | Crossref | Medline | ISI | |
|
Wood, J. A. 2008. Methodology for dealing with duplicate study effects in a meta-analysis. Organizational Research Methods, 11: 79-95. Google Scholar | SAGE Journals | ISI |
