Department of Medicine Connecticut Children's Medical Center Hartford, Connecticut, United States
Background: Currently, patients that cancel or fail to attend an outpatient appointment are considered lost-to-follow-up (LTFU). This method underestimates the true rate of LTFU. Instead of using scheduling data, a better approach is to use the provider’s intent for follow-up, as reflected in the clinical narrative. Objective: Assess the feasibility of Natural Language Processing (NLP) to identify LTFU patients. Design/Methods: We screened a random sample of 4,570 outpatient orthopedic clinic notes from visits occurring between January 1 and July 30 2022 at a tertiary urban medical center. We created an NLP model to identify notes describing follow-up plans. We manually reviewed 10% of the notes (Figure 1), then used 2/3 of the human reviewed data to train an NLP model and 1/3 to validate. We then selected a model threshold that demonstrated sensitivity of 95%, and applied it to all notes, classifying visits as either high or low likelihood of a planned visit. We extracted and linked all hospital follow-up visits up to one year from the index visit (Figure 2). We performed quality assurance for 25% of notes for: A. High-likelihood cases where there was no follow-up (LTFU or false positive); and B. Low-likelihood with follow-up (patient-initiated visits or false negative). Results: Model sensitivity was set a priori at 95%; specificity was 67%. Roughly 2/3 of the patients were assigned to the high likelihood follow-up plan. The majority of high-likelihood index visits had follow-ups (2665/3191, 83%). We manually reviewed 20% of these visits for false positives. Most (103/131, 79%) were LTFU while some (31/103, 30%) were semi-urgent (under 30 days for follow-up). Follow-ups occurred in less than one-fifth of low-likelihood index visits (268/1379, 19.4%). We manually reviewed 20% of these visits for false negatives. Most (47/54, 87%) were not LTFU (i.e. no need to reach out). There were few false negatives (3/54, 5.6%) for semi-urgent visits. Results from follow-up cases are summarized in Table 1.
Conclusion(s): NLP provides a feasible and sensitive method for surveillance of LTFU patients, complementing current systems. Using such a system by creating a dashboard will have high yield to identify patient LTFU, given the high positive predictive value.
