P1311 - Using Natural Language Processing to Improve Automatic Ascertainment of Key Colonoscopy Findings

Monday, October 25, 2021

10:30 AM – 4:15 PM PT

Location: Shoreline Exhibit Hall

Has Audio

Presenting Author(s)

LW

Lei Wang, MS, BMed

University of Texas Southwestern Medical Center
Dallas, TX, United States

Lei Wang, MS, BMed¹, Caitlin C. Murphy, PhD, MPH², Celette S. Skinner, PhD¹, Michael A. Skinner, MD¹, Eric K. Borton, MS¹, Cynthia M. Ortiz, MPH¹, Ellen Hu, MHS¹, Jacquelyn M. Lykken, PhD¹, Ethan A. Halm, MD, MPH, MBA¹
¹University of Texas Southwestern Medical Center, Dallas, TX; ²University of Texas Health Science Center at Houston, Houston, TX

Introduction: Monitoring the quality and findings of colonoscopy is an important component of clinical practice, yet the accuracy of data automatically extracted from endoscopy and pathology reports is suboptimal due to user variation and mix of structured and free text. We developed and validated a Natural Language Processing (NLP) algorithm to improve accuracy of 11 key metrics from colonoscopy.

Methods: We developed a NLP algorithm using machine learning (ML) to improve the accuracy of 5 colonoscopy (indication, bowel preparation quality, cecum reached, any polyp, and polyp removal) and 6 pathology (hyperplastic polyp [HP], any adenoma, tubular adenoma [TA], tubulovillous or villous adenoma [VA], sessile serrated adenoma [SSA], and advanced adenoma [AA]) metrics from colonoscopies performed during 2008 – 2019 at an integrated health system in Dallas, TX. The NLP algorithm used text vectorization, TF-IDF transformation and XGBoost ML with 5-fold cross validation and was trained on a gold standard (e.g., manual chart review). We also compared model performance over two time periods corresponding to different types of endoscopy reports: Provation® MD Gastroenterology (2011-2019) v. legacy dictated reports (2008-2011). We report model accuracy, sensitivity, specificity, and positive/negative predictive values (PPV, NPV).

Results: The NLP algorithm had very high accuracy (Table 1) for: screening indication 99.3%, surveillance indication 99.1%, diagnostic indication 98.7%, good bowel preparation 98.2%, cecum reached 98.3%, any polyp 99.3%, and polyp removed 97.8%. Sensitivity, specificity, PPV and NPV were also very high for these metrics. The NLP algorithm had very high accuracy for pathology-related findings: HP 97.6%, any adenoma 100%, TA 99.4%, VA 97.0%, SSA 98.0%, although accuracy was lower for AA (83.2%). Accuracy was 3.0-9.3% higher when the algorithm was trained on endoscopy reports from Provation compared to legacy reports (trained/validated on reports from the same time period). NLP models trained on data from one time period and applied to another period had lower accuracy (65.5-95.0% for 2011-2019 to 2008-2011 and 59.1-99.8% for 2008-2011 to 2011-2019).

Discussion: Our NLP algorithm achieved very high accuracy for ascertaining 11 key colonoscopy quality and results findings. NLP holds promise for extracting key data from semi-structured Provation endoscopy software and dictated colonoscopy reports.

	Text Data	Outcome	Model Performance Measurements Mean [95% CI]
	Text Data	Outcome	Accuracy ^c	Sensitivity	Specificity	PPV	NPV
Data time period: 2011-2019	Colonoscopy (Provation)	Screening Indication	99.3% [99.2, 99.4]	99.0% [98.8, 99.3]	99.4% [99.3, 99.6]	99.2% [99.0, 99.5]	99.3% [99.1, 99.5]
		Surveillance Indication	99.1% [99.0, 99.1]	98.0% [97.5, 98.4]	99.4% [99.2, 99.5]	97.7% [97.2, 98.2]	99.4% [99.3, 99.6]
		Diagnostic Indication	98.7% [98.6, 98.8]	98.6% [98.3, 98.9]	98.7% [98.5, 99.0]	97.8% [97.3, 98.3]	99.2% [99.0, 99.4]
		Good Bowel Preparation	98.2% [97.6, 98.8]	98.2% [97.5, 98.9]	98.2% [97.4, 99.0]	99.8% [99.7, 99.9]	85.5% [81.0, 89.9]
		Cecum Reached	98.3% [97.8, 98.8]	98.4% [98.1, 98.8]	97.2% [93.7, 100]	99.8% [99.6, 100]	79.8% [75.7, 84.0]
		Any Polyp	99.3% [99.3, 99.4]	99.4% [99.2, 99.6]	99.1% [98.9, 99.4]	99.4% [99.2, 99.6]	99.2% [98.9, 99.4]
		Polyp Removed	97.8% [97.4, 98.2]	97.0% [96.3, 97.6]	98.6% [97.7, 99.6]	98.5% [97.4, 99.5]	97.2% [96.6, 97.8]
	Pathology (EHR)	Hyperplastic Polyp [HP]	97.6% [96.4, 98.9]	97.5% [96.6, 98.4]	97.7% [96.0, 99.3]	95.9% [93.2, 98.7]	98.6% [98.1, 99.1]
		Any Adenoma	100.0% [99.9, 100]	100.0% [100, 100]	100.0% [99.9, 100]	100.0% [99.9, 100]	100.0% [100, 100]
		Tubular Adenoma [TA]	99.4% [99.2, 99.5]	99.3% [99.0, 99.6]	99.5% [99.3, 99.6]	99.6% [99.4, 99.7]	99.1% [98.7, 99.6]
		Villous Adenoma [VA]	97.0% [93.3, 100]	75.3% [66.1, 84.6]	97.3% [93.5, 100]	51.6% [29.2, 73.9]	99.6% [99.5, 99.8]
		Sessile Serrated Adenoma [SSA]	98.0% [95.0, 100]	91.2% [80.0, 100]	98.1% [95.4, 100]	70.2% [43.6, 96.8]	99.8% [99.5, 100]
		Advanced Adenoma [AA] ^b	83.2% [81.2, 85.1]	86.7% [84.0, 89.4]	82.5% [80.1, 84.8]	49.5% [46.4, 52.5]	96.9% [96.3, 97.6]
Data time period: 2008-2011^a	Colonoscopy (Dictated)	Good Bowel Preparation	95.2% [94.4, 96.0]	96.8% [95.8, 97.8]	76.1% [72.2, 79.9]	98.0% [97.8, 98.2]	67.2% [61.7, 72.7]
		Cecum Reached	92.7% [90.6, 94.8]	93.2% [91.3, 95.1]	76.1% [63.8, 88.5]	99.2% [98.8, 99.7]	25.7% [18.6, 32.9]
		Any Polyp	90.0% [89.0, 91.0]	87.5% [84.2, 90.8]	92.1% [89.2, 94.9]	90.2% [87.1, 93.3]	90.1% [88.1, 92.2]
		Polyp Removed	88.8% [88.0, 89.6]	87.6% [83.2, 92.0]	90.0% [87.2, 92.8]	89.5% [86.9, 92.0]	88.6% [85.3, 91.9]
	Pathology (EHR)	Hyperplastic Polyp [HP]	96.8% [96.2, 97.4]	95.5% [93.7, 97.4]	97.4% [95.8, 99.0]	95.1% [92.6, 97.5]	97.8% [96.9, 98.6]
		Any Adenoma	96.6% [96.2, 97.1]	95.6% [95.1, 96.1]	98.1% [97.6, 98.6]	98.7% [98.3, 99.0]	93.9% [92.8, 94.9]
		Tubular Adenoma [TA]	92.1% [90.3, 93.8]	96.5% [95.6, 97.5]	87.1% [84.2, 90.0]	89.3% [87.0, 91.7]	95.7% [94.3, 97.0]
		Advanced Adenoma [AA]	77.9% [73.7, 82.0]	73.5% [67.4, 79.6]	79.7% [73.4, 86.1]	59.7% [52.1, 67.3]	88.5% [86.1, 90.8]
Trained on: 2011-2019 Applied on: 2008-2011	Colonoscopy (Dictated)	Good Bowel Preparation	95.0%	96.6%	76.0%	98.0%	64.8%
		Cecum Reached	85.5%	85.5%	84.2%	99.5%	14.4%
		Any Polyp	80.5%	68.7%	90.1%	85.0%	77.9%
		Polyp Removed	73.7%	70.1%	77.2%	74.6%	73.0%
	Pathology (EHR)	Hyperplastic Polyp [HP]	92.8%	100.0%	89.1%	82.4%	100.0%
		Any Adenoma	94.6%	98.0%	89.6%	93.1%	96.9%
		Tubular Adenoma [TA]	93.4%	97.1%	89.3%	91.0%	96.5%
		Advanced Adenoma [AA]	65.5%	98.6%	52.4%	45.0%	99.0%
Trained on: 2008-2011 Applied on: 2011-2019	Colonoscopy (Provation)	Good Bowel Preparation	97.4%	99.1%	81.1%	98.0%	90.3%
		Cecum Reached	90.3%	91.4%	72.5%	98.1%	35.3%
		Any Polyp	97.2%	96.8%	97.8%	98.4%	95.6%
		Polyp Removed	97.1%	95.7%	98.4%	98.2%	96.1%
	Pathology (EHR)	Hyperplastic Polyp [HP]	86.2%	62.9%	99.0%	97.3%	82.9%
		Any Adenoma	99.8%	100.0%	99.6%	99.7%	100.0%
		Tubular Adenoma [TA]	99.4%	99.4%	99.3%	99.4%	99.3%
		Advanced Adenoma [AA]	59.1%	98.1%	51.4%	28.4%	99.3%

Table: Table 1: NLP Model Performance for Each Outcome Across Four Scenarios.

Footnote:
a Due to limitation of gold standard availability and sample size, partial of the variables are excluded for 2008-2011 time period models.
b Advanced adenoma is defined as adenoma with villous growth pattern, high-grade dysplasia, any adenoma ≥10 mm in size, or 3 or more of any adenomas of any size or histology.
c Accuracy is defined as number of correct classifications/number of sample size.

Disclosures:

Lei Wang indicated no relevant financial relationships.

Caitlin Murphy: Freenome – Consultant.

Celette Skinner indicated no relevant financial relationships.

Michael Skinner indicated no relevant financial relationships.

Eric Borton indicated no relevant financial relationships.

Cynthia Ortiz indicated no relevant financial relationships.

Ellen Hu indicated no relevant financial relationships.

Jacquelyn Lykken indicated no relevant financial relationships.

Ethan Halm indicated no relevant financial relationships.

Lei Wang, MS, BMed¹, Caitlin C. Murphy, PhD, MPH², Celette S. Skinner, PhD¹, Michael A. Skinner, MD¹, Eric K. Borton, MS¹, Cynthia M. Ortiz, MPH¹, Ellen Hu, MHS¹, Jacquelyn M. Lykken, PhD¹, Ethan A. Halm, MD, MPH, MBA¹. P1311 - Using Natural Language Processing to Improve Automatic Ascertainment of Key Colonoscopy Findings, ACG 2021 Annual Scientific Meeting Abstracts. Las Vegas, Nevada: American College of Gastroenterology.