Fabrication and errors in the bibliographic citations generated by ChatGPT

Although chatbots such as ChatGPT can facilitate cost-effective text generation and editing, factually incorrect responses (hallucinations) limit their utility. This study evaluates one particular type of hallucination: fabricated bibliographic citations that do not represent actual scholarly works. We used ChatGPT-3.5 and ChatGPT-4 to produce short literature reviews on 42 multidisciplinary topics, compiling data on the 636 bibliographic citations (references) found in the 84 papers. We then searched multiple databases and websites to determine the prevalence of fabricated citations, to identify errors in the citations to non-fabricated papers, and to evaluate adherence to APA citation format. Within this set of documents, 55% of the GPT-3.5 citations but just 18% of the GPT-4 citations are fabricated. Likewise, 43% of the real (non-fabricated) GPT-3.5 citations but just 24% of the real GPT-4 citations include substantive citation errors. Although GPT-4 is a major improvement over GPT-3.5, problems remain.

• Publication year

  2023

• Venue

  Scientific Reports

• Publication date

  2023-09-07

• Fields of study

  Medicine, Computer Science

• Identifiers
  DOI 10.1038/s41598-023-41032-5PMID 37679503PMCID 10484980
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors

  2023cited by this paper
• Can ChatGPT Help in Electronics Research and Development? A Case Study with Applied Sensors

  2023cited by this paper
• High Rates of Fabricated and Inaccurate References in ChatGPT-Generated Medical Content

  2023influential reference
• In Reference to “Role of Chat GPT in Public Health”, to Highlight the AI’s Incorrect Reference Generation

  2023cited by this paper
• Quality of citation data using the natural language processing tool ChatGPT in rheumatology: creation of false references

  2023cited by this paper
• Testing of detection tools for AI-generated text

  2023cited by this paper
• MRI-Based End-To-End Pediatric Low-Grade Glioma Segmentation and Classification

  2023cited by this paper
• A cardiologist-like computer-aided interpretation framework to improve arrhythmia diagnosis from imbalanced training datasets

  2023cited by this paper
• ChatGPT and the potential growing of ghost bibliographic references

  2023cited by this paper
• A Conversation on Artificial Intelligence, Chatbots, and Plagiarism in Higher Education

  2023cited by this paper
• Tools such as ChatGPT threaten transparent science; here are our ground rules for their use

  2023cited by this paper
• ChatGPT at Universities – The Least of Our Concerns

  2023cited by this paper
• Artificial Hallucinations in ChatGPT: Implications in Scientific Writing

  2023cited by this paper
• The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation With ChatGPT and a Call for Papers

  2023cited by this paper
• Using ChatGPT for language editing in scientific articles

  2023cited by this paper
• ChatGPT in education: Strategies for responsible implementation

  2023cited by this paper
• ChatGPT and a new academic reality: Artificial Intelligence‐written research papers and the ethics of the large language models in scholarly publishing

  2023cited by this paper
• GPT-4 Technical Report

  2023cited by this paper
• Intrapartum amnioinfusion reduces meconium aspiration syndrome and improves neonatal outcomes in patients with meconium-stained fluid: a systematic review and meta-analysis.

  2023cited by this paper
• Learning to Fake It: Limited Responses and Fabricated References Provided by ChatGPT for Medical Questions

  2023cited by this paper
• To ChatGPT, or not to ChatGPT: That is the question!

  2023cited by this paper
• GPT detectors are biased against non-native English writers

  2023cited by this paper
• Beware of References when Using ChatGPT as a Source of Information to Write Scientific Articles.

  2023cited by this paper
• Evaluating AIGC Detectors on Code Content

  2023cited by this paper
• Exploring the Boundaries of Reality: Investigating the Phenomenon of Artificial Intelligence Hallucination in Scientific Writing Through ChatGPT References

  2023cited by this paper
• A Preliminary Investigation of Fake Peer-Reviewed Citations and References Generated by ChatGPT

  2023cited by this paper
• Accuracy of Information and References Using ChatGPT-3 for Retrieval of Clinical Radiological Information

  2023cited by this paper
• Training language models to follow instructions with human feedback

  2022cited by this paper
• AI bot ChatGPT writes smart essays - should professors worry?

  2022cited by this paper
• Two-year-olds are vigilant of others' non-verbal cues to credibility.

  2010cited by this paper
• Detection of Deception

  2007cited by this paper
• An ego-centric citation analysis of the works of Michael O. Rabin based on multiple citation indexes

  2006cited by this paper
• Deflated, inflated and phantom citation counts

  2006cited by this paper
• As we may search : Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases

  2005cited by this paper
• Intuitive evaluation of likelihood judgment producers: evidence for a confidence heuristic

  2004cited by this paper

• A Conditional Companion: Lived Experiences of People with Mental Health Disorders Using LLMs

  2026cites this paper
• Deep research capabilities in GPT-5 thinking and Gemini 2.5 Pro improve citation integrity and concordance with American Academy of Orthopaedic Surgeons anterior cruciate ligament and rotator cuff guidelines.

  2026cites this paper
• Confabulated references in the age of AI: contamination of the biomedical scientific literature

  2026cites this paper
• Do large language models know basic facts about journal articles?

  2026cites this paper
• Rethink literature review of design research in an age of AI – from ‘secretary work’ to scholarly synthesis of insight, frameworks, and foresight

  2026cites this paper
• BibAgent: An Agentic Framework for Traceable Miscitation Detection in Scientific Literature

  2026cites this paper
• Pondering the Future of Chemical Research amid the Wider Adoption of Artificial Intelligence Technologies.

  2026cites this paper
• From essay to AI-augmented writing: reframing pedagogy and feedback through the AAWP model

  2026cites this paper
• Authors, Academics, and AI: Questions About Research and Publishing in the World of Artificial Intelligence

  2026cites this paper
• Performance of a Small Language Model Versus a Large Language Model in Answering Glaucoma Frequently Asked Patient Questions: Development and Usability Study

  2026cites this paper
• Medical large language models and systems in the clinical application of spinal diseases: Current status, challenges, and future prospects.

  2026cites this paper
• Guest editorial introduction to the special issue on “Writing and Editing Scientific Manuscripts in the Era of Generative Artificial Intelligence: Roles and Responsibilities of Editors, Authors, and Reviewers”

  2026cites this paper
• Artificial Intelligence in Medical Education: Transformative Potential, Current Applications, and Future Implications.

  2026cites this paper
• Assessment of ChatGPT-5 as an Artificial Intelligence Tool for Exploring Emerging Dimensions of Clinical Simulation: A Proof-of-concept Study

  2026cites this paper
• Large Language Models for High-Entropy Alloys: Literature Mining, Design Orchestration, and Evaluation Standards

  2026influential citation
• Scholarly Contributions at the Intersection of Artificial Intelligence and an Evolving Research Landscape

  2026cites this paper
• When References Mislead: Verification, AI Attribution, and Academic Bullying in Scholarly Evaluation

  2026cites this paper
• Limitations and mitigation strategies for using generative artificial intelligence in medical writing: a narrative review

  2026cites this paper
• Large language models in trauma anesthesia education.

  2026cites this paper
• Artificial Intelligence for Academic Text Generation in Analytical Chemistry: Current Risks, Indicators, and Perspectives toward Greener and More Sustainable Approaches.

  2026cites this paper
• Citing the Literature

  2026cites this paper
• MotionPhysics: Learnable Motion Distillation for Text-Guided Simulation

  2026cites this paper
• ChatGPT-4o Mini Fabricates and Miscites Evidence for American Academy of Orthopaedic Surgeons Hip Fracture Clinical Practice Guidelines

  2026cites this paper
• Ethical Issues in AI-Generated Texts: A Systematic Review and Analysis

  2026influential citation
• Artificial Intelligence Chatbots Taking American Board of Endodontics Simulated Oral Board Examination.

  2026cites this paper
• How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations

  2026influential citation
• Consensus on the Application of Generative Artificial Intelligence in Medical Manuscript Writing

  2026cites this paper
• Why every scientist needs a librarian

  2026cites this paper
• Query Augmented Generation (QAG) from the Genomic Data Commons for Accurate Variant Statistics

  2025cites this paper
• CiteGuard: Faithful Citation Attribution for LLMs via Retrieval-Augmented Validation

  2025cites this paper
• Benchmarking generative AI tools for literature retrieval and summarization in genomic variant interpretation

  2025cites this paper
• Quality and readability of chatbot responses to patient questions: A systematic cross-sectional meta-synthesis

  2025cites this paper
• College Students’ Information Literacy Levels and Their Usage of ChatGPT in Support of Learning

  2025cites this paper
• AI performance in emergency medicine fellowship examination: comparative analysis of ChatGPT-4o, Gemini 2.0, Claude 3.5, and DeepSeek R1 models

  2025cites this paper
• Fake scientific journals are here to stay

  2025cites this paper
• Generating Transparency

  2025cites this paper
• Fabricated or accurate? Ethical concerns and citation hallucination in aI-generated scientific writing on musculoskeletal topics

  2025cites this paper
• Figurative Language in Sabian Poetry by Oka Rusmini

  2025cites this paper
• The Prompt Engineering Report Distilled: Quick Start Guide for Life Sciences

  2025cites this paper
• Generative AI and the Information Society: Ethical Reflections from Libraries

  2025cites this paper
• Assessing the Reliability of ChatGPT and Gemini in Identifying Relevant Orthodontic Literature

  2025cites this paper
• Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Models

  2025cites this paper
• Large language models in peer review: challenges and opportunities

  2025cites this paper
• The Paradox of Ethical AI-Assisted Research

  2025cites this paper
• How does the credibility of vaccine information compare across traditional search engines and AI-based conversational agents?

  2025cites this paper
• Artificial Intelligence in Research: A Double-Edged Sword in Evidence Generation

  2025cites this paper
• The Role of AI in Historical Simulation Design: A TPACK Perspective on a French Revolution Simulation Design Experience

  2025cites this paper
• Assessment of Deep Research for dermatology literature reviews: Deep concern over the hype.

  2025cites this paper
• Generative chatbots in headache education and research: A narrative review

  2025cites this paper
• Co-intelligent Design of Catalysis Research with Large Language Models: Hype or Reality?

  2025cites this paper
• Solar Radiation Estimation Using an AI-Driven Empirical Model: Calibration and Evaluation

  2025cites this paper
• Human-AI Narrative Synthesis to Foster Shared Understanding in Civic Decision-Making

  2025cites this paper
• Generative Artificial Intelligence and Collaboration: Exploring Religious Human-Machine Communication and Tensions in Leadership Practices

  2025cites this paper
• Factors complicating the identification and processing of duplicates in bibliographic records: A theoretical perspective

  2025cites this paper
• Exploring the scope of generative AI in literature review development

  2025influential citation
• Student Perspectives on Using Generative Artificial Intelligence for Research: A Qualitative Approach

  2025cites this paper
• The Pediatric Surgeon's AI Toolbox: How Large Language Models Like ChatGPT Are Simplifying Practice and Expanding Global Access

  2025cites this paper
• Transformation of Islamic values in the era of artificial intelligence

  2025cites this paper
• Evaluating the Accuracy of Responses by Large Language Models for Information on Disease Epidemiology

  2025cites this paper
• Creative personal identity in the age of generative AI: A social-cognitive pathway of AI literacy, self-efficacy, and mindset

  2025cites this paper
• Artificial Intelligence Performance in Introductory Biology: Passing Grades but Poor Performance at High Cognitive Complexity

  2025cites this paper
• Using Artificial Intelligence for Scholarly Writing

  2025influential citation
• TreeReader: A Hierarchical Academic Paper Reader Powered by Language Models

  2025cites this paper
• Generative Artificial Intelligence and Language Teaching

  2025cites this paper
• Explainable Information Retrieval in the Audit Domain

  2025cites this paper
• Capabilities of GPT-5 across critical domains: Is it the next breakthrough?

  2025cites this paper
• AI Versus Human Feedback in Mixed Reality Simulations: Comparing LLM and Expert Mentoring in Preservice Teacher Education on Controversial Issues

  2025cites this paper
• Research on Artificial Intelligence in Libraries

  2025cites this paper
• Evaluating Large Language Models for Gene-to-Phenotype Mapping: The Critical Role of Full-Text Database Access

  2025cites this paper
• GPT Editors, Not Authors: The Stylistic Footprint of LLMs in Academic Preprints

  2025cites this paper
• The role of moral intention and moral obligation in predicting attitudes toward avoiding AIgiarism: A protection motivation theory perspective

  2025cites this paper
• A Study of Graduate Students’ Experiences of Artificial Intelligence at the University of New Brunswick

  2025cites this paper
• Computerized diagnostic decision support systems—Isabel Pro versus ChatGPT-4 part II

  2025cites this paper
• Generative AI and academic scientists in US universities: Perception, experience, and adoption intentions

  2025cites this paper
• Algorithm characteristics, perceived credibility and verification of ChatGPT-generated content: a moderated nonlinear model

  2025cites this paper
• Confabulation dynamics in a reservoir computer: Filling in the gaps with untrained attractors

  2025cites this paper
• Exploring the change in scientific readability following the release of ChatGPT

  2025cites this paper
• The potential of artificial intelligence to transform medicine

  2025cites this paper
• Comparative Diagnostic Accuracy of ChatGPT-4 and Machine Learning in Differentiating Spinal Tuberculosis and Spinal Tumors.

  2025cites this paper
• LITERAS: Biomedical literature review and citation retrieval agents

  2025cites this paper
• Quality in science communication with communicative artificial intelligence: A principle-based framework

  2025cites this paper
• Critically Reading Science-Related Texts Produced by ChatGPT

  2025influential citation
• The Integration of Generative AI Tools in Academic Writing: Implications for Student Research

  2025cites this paper
• False authorship: an explorative case study around an AI-generated article published under my name

  2025cites this paper
• AI on the Shoulders of Giants: Using Kuhlthau’s Information Search Process to Improve AI Support for Information-Seeking

  2025cites this paper
• Generative AI in Undergraduate Education: An Early View of Developments, Prospects, and Challenges of the AI Revolution

  2025cites this paper
• Writing is thinking

  2025cites this paper
• Opinion: The need for higher education in lighting: Human insight in a digital age

  2025cites this paper
• Assessment of factors influencing the citation level of scientific publications in the field of sport and physical activity

  2025cites this paper
• Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models

  2025cites this paper
• Responsible use of AI in social science research

  2025cites this paper
• Comparison of Generative Artificial Intelligence and Student-Generated Veterinary Handouts.

  2025cites this paper
• Beyond human touch: evaluating the effectiveness of AI, human, and hybrid-generated tourism promotional texts

  2025cites this paper
• Responsible integration of generative artificial intelligence in academic writing: a narrative review and synthesis

  2025cites this paper
• Evaluating AI performance in nephrology triage and subspecialty referrals

  2025cites this paper
• How should the advancement of large language models affect the practice of science?

  2025cites this paper
• Intention to use ChatGPT among law educators in Saudi Arabia

  2025cites this paper
• Modeling teacher education students’ adoption of large language models through an extended technology acceptance framework

  2025cites this paper
• AI-driven fabrication of healthcare survey data: methods, motivations, and ethical implications

  2025cites this paper
• Generative AI Literacy: A Comprehensive Framework for Literacy and Responsible Use

  2025cites this paper