ChatGPT Hallucinates Non-existent Citations: Evidence from Economics
Abstract
In this study, we generate prompts derived from every topic within the Journal of Economic Literature to assess the abilities of both GPT-3.5 and GPT-4 versions of the ChatGPT large language model (LLM) to write about economic concepts. ChatGPT demonstrates considerable competency in offering general summaries but also cites non-existent references. More than 30% of the citations provided by the GPT-3.5 version do not exist and this rate is only slightly reduced for the GPT-4 version. Additionally, our findings suggest that the reliability of the model decreases as the prompts become more specific. We provide quantitative evidence for errors in ChatGPT output to demonstrate the importance of LLM verification.
JEL Codes: B4; O33; I2
Get full access to this article
View all access and purchase options for this article.
References
Alkaissi H., McFarlane S. I. (2023). Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus, 15(2), e35179. https://doi.org/10.7759/cureus.35179
Buchanan J., Hickman W. (2023). Do people trust humans more than ChatGPT? https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4632842
Bybee L. (2023). Surveying Generative AI’s Economic Expectations. https://arxiv.org/abs/2305.02823
Cowen T., Tabarrok A. T. (2023). How to Learn and Teach Economics with Large Language Models, Including GPT (GMU Working Paper in Economics No. 23-18). https://doi.org/10.2139/ssrn.4391863
Eloundou T., Manning S., Mishkin P., Rock D. (2023). GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models. https://doi.org/10.48550/arXiv.2303.10130
Geerling W., Mateer G. D., Wooten J., Damodaran N. (2023). ChatGPT has aced the test of understanding in college economics: Now what? The American Economist, 68(2), 233–245. https://doi.org/10.1177/05694345231169654
Goldfarb A., Taska B., Teodoridis F. (2023). Could machine learning be a general purpose technology? A comparison of emerging technologies using data from online job postings. Research Policy, 52(1), 104653. https://doi.org/10.1016/j.respol.2022.104653
Horton J. J. (2023). Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? National Bureau of Economic Research. https://www.nber.org/papers/w31122
Ji Z., Lee N., Frieske R., Yu T., Su D., Xu Y., Ishii E., Bang Y. J., Madotto A., Fung P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1–38. https://doi.org/10.1145/3571730
Jungherr A. (2023). Using ChatGPT and Other Large Language Model (LLM) Applications for. https://fis.uni-bamberg.de/handle/uniba/58950
Korinek A. (2023). Language Models and Cognitive Automation for Economic Research (NBER Working Paper No. 30957). National Bureau of Economic Research. https://doi.org/10.3386/w30957
OpenAI. (2022). Introducing ChatGPT. OpenAI Blog. https://openai.com/blog/chatgpt
OpenAI. (2023). GPT-4. OpenAI Blog. https://openai.com/research/gpt-4
Peng S., Kalliamvakou E., Cihon P., Demirer M. (2023). The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. https://doi.org/10.48550/arXiv.2302.06590
Roberts J., Lüddecke T., Das S., Han K., Albanie S. (2023). GPT4GEO: How a Language Model Sees the World’s Geography. http://arxiv.org/abs/2306.00020
Thompson S. A. (2023). A.I.-Generated Content Discovered on News Sites, Content Farms and Product Reviews. The New York Times. https://www.nytimes.com/2023/05/19/technology/ai-generated-content-discovered-on-news-sites-content-farms-and-product-reviews.html
Biographies
Joy Buchanan, an Associate Professor at Samford University’s Brock School of Business. Her research focuses on applying the experimental method to better understand behavior in labor markets and institutions. She has designed an experiment to test willingness to do computer programming.
Stephen Hill, holding a Ph.D. in Operations Management from the University of Alabama, serves as an Associate Professor of Data Analytics at Samford University. His application of analytics techniques is demonstrated through research spanning diverse sectors, including sports, supply chains, education, and health care.
Olga Shapoval serves as an instructor of economics and data analytics at Samford University. She earned her Ph.D. in Economics at the University of Nevada in Reno, specializing in policy evaluation within the health and education sectors.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published online: November 23, 2023
Issue published: March 2024
Keywords
Authors
Metrics and citations
Metrics
Article usage*
Total views and downloads: 817
*Article usage tracking started in December 2016
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 0
Crossref: 8
- Hallucination in AI-generated financial literature reviews: evaluating bibliographic accuracy
- When AI Gives Bad Advice: Critical thinking in human-AI collaborations
- ‘As of my last knowledge update’: How is content generated by ChatGPT infiltrating scientific papers published in premier journals?
- Do people trust humans more than ChatGPT?
- Navigating AI in Academic Libraries
- ChatGPT in Undergraduate Education: Performance of GPT-3.5 and Identification of AI-Generated Text in Introductory Neuroscience
- ChatGPT und halluzinierte Referenzen in Artikeln aus ausgewählten Bereichen der Betriebswirtschaftslehre
- Generative AI and the Future of Democratic Citizenship
- Ruled by robots: preference for algorithmic decision makers and perceptions of their choices
- Chatbot or Humanaut? How the Source of Advice Impacts Behavior in One-shot Social Dilemmas
Figures and tables
Figures & Media
Tables
View Options
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
I am signed in as:
View my profileSign out
I can access personal subscriptions, purchases, paired institutional access and free tools such as favourite journals, email alerts and saved searches.
loading institutional access options
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
