A Comprehensive Benchmark Study on Biomedical Text Generation and Mining with ChatGPT

In recent years, the development of natural language process (NLP) technologies and deep learning hardware has led to significant improvement in large language models(LLMs). The ChatGPT, the state-of-the-art LLM built on GPT-3.5, shows excellent capabilities in general language understanding and reasoning. Researchers also tested the GPTs on a variety of NLP related tasks and benchmarks and got excellent results. To evaluate the performance of ChatGPT on biomedical related tasks, this paper presents a comprehensive benchmark study on the use of ChatGPT for biomedical corpus, including article abstracts, clinical trials description, biomedical questions and so on. Through a series of experiments, we demonstrated the effectiveness and versatility of Chat-GPT in biomedical text understanding, reasoning and generation.

• Publication year

  2023

• Venue

  bioRxiv

• Publication date

  2023-04-20

• Fields of study

  Biology, Medicine, Computer Science

• Identifiers
  DOI 10.1101/2023.04.19.537463
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• LinkBERT: Pretraining Language Models with Document Links

  2022cited by this paper
• Training language models to follow instructions with human feedback

  2022cited by this paper
• SciFive: a text-to-text transformer model for biomedical literature

  2021cited by this paper
• Results of the Seventh Edition of the BioASQ Challenge

  2020cited by this paper
• Language Models are Few-Shot Learners

  2020cited by this paper
• PubMedQA: A Dataset for Biomedical Research Question Answering

  2019cited by this paper
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

  2019cited by this paper
• Language Models are Unsupervised Multitask Learners

  2019cited by this paper
• Improving Language Understanding by Generative Pre-Training

  2018cited by this paper
• A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature

  2018cited by this paper
• BIOSSES: a semantic sentence similarity estimation system for the biomedical domain

  2017cited by this paper
• A neural network multi-task learning approach to biomedical named entity recognition

  2017cited by this paper
• BioCreative V CDR task corpus: a resource for chemical disease relation extraction

  2016cited by this paper
• Automatic semantic classification of scientific literature according to the hallmarks of cancer

  2016cited by this paper
• NCBI disease corpus: A resource for disease name recognition and concept normalization

  2014cited by this paper
• Extraction of relations between genes and diseases from text and large-scale data analysis: implications for translational research

  2014cited by this paper
• The DDI corpus: An annotated corpus with pharmacological substances and drug-drug interactions

  2013cited by this paper
• Exploring the Limits

  2009cited by this paper
• Overview of BioCreative II gene mention recognition

  2008cited by this paper
• Evaluation of PICO as a Knowledge Representation for Clinical Questions

  2006cited by this paper
• Introduction to the Bio-entity Recognition Task at JNLPBA

  2004cited by this paper

• Active Learning in Biomedical Text Classification Using a Bag-of-Regular-Expressions Approach

  2025cites this paper
• Large language models help facilitate the automated synthesis of information on potential pest controllers

  2024cites this paper
• Enhancing systematic reviews in orthodontics: a comparative examination of GPT-3.5 and GPT-4 for generating PICO-based queries with tailored prompts and configurations.

  2024cites this paper
• A Novel Cascade Instruction Tuning Method for Biomedical NER

  2024cites this paper
• Utilizing Large Language Models for Named Entity Recognition in Traditional Chinese Medicine against COVID-19 Literature: Comparative Study

  2024cites this paper
• Simple and effective embedding model for single-cell biology built from ChatGPT

  2024cites this paper
• Comparative Performance Evaluation of Large Language Models for Extracting Molecular Interactions and Pathway Knowledge

  2023cites this paper
• LitSumm: large language models for literature summarization of noncoding RNAs

  2023cites this paper
• A Survey of Large Language Models in Medicine: Progress, Application, and Challenge

  2023cites this paper
• nach0: multimodal natural and chemical languages foundation model

  2023cites this paper
• Foundation and large language models: fundamentals, challenges, opportunities, and social impacts

  2023cites this paper
• Iterative Prompt Refinement for Mining Gene Relationships from ChatGPT

  2023cites this paper
• Evaluating ChatGPT's Performance for Multilingual and Emoji-based Hate Speech Detection

  2023cites this paper
• ChatGPT, Bard, and Large Language Models for Biomedical Research: Opportunities and Pitfalls

  2023cites this paper
• Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health

  2023cites this paper
• GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT

  2023cites this paper
• Using ChatGPT to Generate Gendered Language

  2023cites this paper
• Benchmarking large language models for biomedical natural language processing applications and recommendations

  2023cites this paper
• Unveiling the Reliability of ChatGPT Answers in the Biomedical Realm: An Assessment in the

  2023cites this paper
• Developing a Deep Learning Model for Text Generation from a Given Set of Text

  2023cites this paper
• ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing

  2023cites this paper
• A systematic evaluation of large language models for biomedical natural language processing: benchmarks, baselines, and recommendations

  year unknowncites this paper
• G ENE PT: A S IMPLE B UT H ARD - TO -B EAT F OUNDATION M ODEL FOR G ENES AND C ELLS B UILT F ROM C HAT GPT

  year unknowncites this paper
• opportunities and limitations

  year unknowncites this paper