Artificial Intelligence in Neurosurgery: Examining ChatGPT's Performance on the ABNS Primary Exam

Introduction

Artificial intelligence has the potential to revolutionize the healthcare sector by improving diagnostic accuracy, reducing costs, and increasing access to care. However, due to a variety of technical, ethical, and regulatory challenges, the use of AI in healthcare remains limited. ChatGPT (OpenAI Inc., San Francisco, CA), a large language model (LLM), has received attention for its ability to perform various natural language processing tasks. To gain a better understanding of the feasibility of using artificial intelligence in the field of neurosurgery, this study evaluates ChatGPT’s potential use in neurosurgery education by testing its clinical reasoning on simulated questions for the Primary Neurosurgery Written Board Exam.

Objectives

We aim to evaluate ChatGPT's performance on the CNS Self-Assessment Neurosurgery Exam (SANS) Board Review Prep Examination in order to better understand the potential future applications of large language models in the realm of medical education, specifically in the field of neurosurgery.

Methods

A total of 99 questions were obtained from the CNS Self-Assessment Neurosurgery Exam (SANS) Board Review Prep Exam 1. They were grouped by neurosurgical topics, including Functional (10), Fundamentals (9), Neuropathology (7), Neuroradiology (3), Pain (10), Pediatrics (10), Peripheral Nerve (10), Spine (10), Trauma (10), Tumor (10), and Vascular (10). In the first round of analysis, ChatGPT was presented with the question and multiple-choice answers verbatim from SANS. In the second round of analysis, ChatGPT was presented with the question only, requiring the model to provide an explanation with its response. Concordance for these answer explanations was assessed by a board-certified attending neurosurgeon and residents to detect any errors or contradictions within the explanation.

Results

The ChatGPT model answered 99 questions with accuracy of 54.5% on multiple-choice and 45.5% on free response explanation components, having concordance on 77 questions (77.8%), indicating a low level of contradictions in the explanation. On subgroup analysis, the model had higher averages for Functional (90%), Fundamentals (77.8%), Peripheral Nerve (70%), Spine (70%), and Tumor (70%) on multiple-choice and higher averages for Functional (60%), Pediatrics (50%), Peripheral Nerve (80%), Spine (50%), and Tumor (80%) on free response explanation. The ChatGPT model exceeded the concordance average for Functional (90%), Pain (90%), Peripheral Nerve (100%), Spine (90%), and Trauma (100%).

Conclusions

Our study sheds light on the impressive capabilities and potential future applications of LLMs in the realm of education, specifically in the field of neurosurgery. The ChatGPT model performed well in several neurosurgical topics, including functional, peripheral nerve, and tumor. Although ChatGPT did not achieve last year’s pass mark for the neurosurgery boards of 61-64% on MCQ and EXP, the neurosurgery evaluators noted the depth and utility of response provided by ChatGPT in its explanations, underscoring the significant potential of machine learning in medicine. A more informed and accurate LLM could provide tremendous utility for the neurosurgical field. Further research is indicated to continue exploring the capabilities of ChatGPT as well as facilitating development of similar LLMs specific to medical and neurosurgical domains.