1358 - Using natural language processing to optimize case ascertainment of acute otitis media in a large, state-wide pediatric practice network

Background: Antibiotics are the most commonly prescribed drugs for children and frequently inappropriately prescribed. Outpatient antimicrobial stewardship interventions aim to reduce inappropriate antibiotic use. Previous work has relied on diagnosis coding for case identification which may be inaccurate. In this study, we sought to develop automated methods for analyzing note text to identify cases of acute otitis media (AOM) based on clinical documentation.

Methods: We conducted a cross-sectional retrospective chart review and sampled encounters from 7/1/2018 – 6/30/2019 for patients < 5 years old presenting for a problem-focused visit. Complete note text and limited structured data were extracted for 12 randomly selected weekdays (one from each month during the study period). An additional weekday was randomly selected for validation. The primary outcome was correctly identifying encounters where AOM was present. Human review was considered the “gold standard” and was compared to ICD codes, a natural language processing (NLP) model, and a recursive partitioning (RP) model.

Results: A total of 2,724 encounters were included in the training cohort and 793 in the validation cohort. ICD codes and NLP had good performance overall with sensitivity 91.2% and 93.1% respectively in the training cohort. However, NLP had a significant drop-off in performance in the validation cohort (sensitivity: 83.4%). The RP model had the highest sensitivity (97.2% training cohort; 94.1% validation cohort) out of the 3 methods.Figure 1. Details of encounters included in the training and validation cohorts.
Table 1. Performance of ICD coding, a natural language processing (NLP) model, and a recursive partitioning (RP) model for identifying cases of acute otitis media (AOM)


Conclusion: Natural language processing of outpatient pediatric visit documentation can be used successfully to create models accurately identifying cases of AOM based on clinical documentation. Combining NLP and structured data can improve automated case detection, leading to more accurate assessment of antibiotic prescribing practices. These techniques may be valuable in optimizing outpatient antimicrobial stewardship efforts.