SCIENCE@home

Descriptive content analysis assessment of ChatGPT responses to substance use disorder treatment questions compared to National health guidelines.

Morgan Decker,Christine Kamm,Sara Burgoa,Meera Rao,M. Mejia,C. Ramdin,Adrienne Dean,Mélodie A Nasr,Lewis S. Nelson,Lea Sacca

Published 2026 in Drug and Alcohol Dependence

ABSTRACT

BACKGROUND Artificial intelligence (AI)-powered large language models like ChatGPT are increasingly used by the public to access health information. These platforms may be particularly appealing for high-risk conditions such as substance use disorder (SUD), where anonymity and nonjudgmental responses are valued. Despite growing interest in AI-assisted health education, limited research has assessed the quality of ChatGPT's content when it comes to accuracy and completeness on complex behavioral health topics. This study evaluated the accuracy and clinical consistency of ChatGPT's responses to SUD-related questions compared to national health guidelines. METHODS This descriptive study, using a content analysis approach, analyzed ChatGPT 3.5's and 5's responses to 14 clinically relevant SUD-related questions, drawn from over 200 FAQs sourced from six leading U.S. health organizations in comparison to the top SUD questions asked by US adults using ChatGPT. Each response was independently assessed by a multidisciplinary team for accuracy, clarity, and appropriateness using an evidence-informed rating system. Responses were categorized as excellent, satisfactory requiring minimal clarification, satisfactory requiring moderate clarification, or unsatisfactory. Discrepancies were resolved through consensus. RESULTS Among the 14 responses, 3 were rated excellent, 9 were satisfactory requiring minimal clarification, and 2 were satisfactory requiring moderate clarification. None were rated unsatisfactory. ChatGPT responses were generally accurate for straightforward questions but lacked clinical nuance and specificity in more complex scenarios, particularly regarding individualized care recommendations, withdrawal management, and treatment planning. CONCLUSION As AI becomes more integrated into health information-seeking behaviors, continued evaluation of its role and potential impact in addiction medicine is essential.

PUBLICATION RECORD

CITATION MAP

EXTRACTION MAP

CLAIMS

  • No claims are published for this paper.

CONCEPTS

  • No concepts are published for this paper.

REFERENCES

Showing 1-35 of 35 references · Page 1 of 1

CITED BY

  • No citing papers are available for this paper.

Showing 0-0 of 0 citing papers · Page 1 of 1