In this subsection, we will explore related work. We categorize related work into two categories: 1) work measuring wait-times in healthcare system settings, and 2) work performing text classification on referral notes or other types of clinical notes.

Traditionally, work measuring wait-times in healthcare systems do not engage in machine learning prediction to enable the measurement. This is because either the required labels can be found within the administrative data [12, 13], or because they have used manual effort to label the notes [11].

There is work which develops machine learning applications that reduce wait-times [14], or attempt to predict length-of-stay [15] (which is related to patient wait-times, though in a referral setting–the focus of our work). Gonçalves et al. [16] attempted to predict patient wait-times between triage to observation. Lin et al. [17] attempt to predict wait-times for patients for Pediatric Ophthalmology outpatient clinic using patient characteristics and other relevant features. Outside of the medical domain (e.g., business or government offices) other work has built models to predict expected wait-times given historical data [18, 19].

We have been unable to find other works which used machine learning on free text clinical notes to label the target specialty of referral notes to enable wait-time estimation–most machine learning work attempts to predict the wait-time directly. Bauder et al. [20] attempted to classify medical provider specialties from their billed codes (a different mode of input) to detect fraud (a different goal).

There is a large body of work applying machine learning to clinical notes to perform classification. Amongst the many classification tasks are identification of patients with pathologies [21], prediction of admission [22], and predicting high-cost users [23] among many other tasks (e.g., medical subdomain classification [24]). To perform the classification these works train and evaluate a variety of machine learning models such as: Logistic Regression [21–23], Random Forests [21, 22], Gradient-boosted models [22], Support Vector Machines [25], Shallow Neural Networks [21], and deep neural networks [22]. The results of which model performs better is heavily dependent on the dataset and task at hand.

The machine learning pipeline introduced in this section is developed and evaluated on the subset of data described above using the Python package Scikit-learn [29]. The machine learning pipeline follows these steps:

First, we explore the relationship between number of training examples and performance. Fig 2 plots the F₁Score on the y-axis and the number of examples on the x-axis. We observe that more training examples generally means better performance. However, for certain specialties (e.g., nephrology), good performance was achieved without many examples. On the other hand, there are specialties with many examples (i.e., notes) that perform poorly (when compared to other specialties with a similar number of notes), such as general surgery.

These observations can likely be explained by examining the uniqueness of a specialty. A specialty that examines the same body parts, and shares procedures and terms relevant to other specialties is likely to be harder to classify than specialties that overlap less with other specialties (e.g., OGBYN).

In this section, we examined the relationships between classification performance (accuracy) and length of referral note (as measured by number of tokens). Fig 3 plots the number of tokens in a referral note against the average accuracy of models on referral notes averaged across all specialties. For the best model of each specialty, we individually calculated the accuracy of the model on all the notes of a particular length (i.e., number of tokens). For example, if there are 5 classifiers and 10 notes of a particular size: we would separately calculate the accuracy of each of those classifiers on the 10 notes (resulting in 10 accuracy values). We then aggregated these accuracy measurements for each number of tokens by averaging the accuracy of all classifiers for each length (i.e., each presented value would be the average of the accuracy of all 5 classifier in the example). Values of “number of tokens” that had fewer than 5 associated notes are not plotted (in accordance with ICES privacy guidelines). See S1 Appendix which provides the average accuracies associated with each note length.

We observe that having very few tokens (e.g., <10) or very many tokens (e.g., >125) results in sub-optimal performance (when compared to notes having between 50 and 100 tokens). We observe, from manual examination of the notes, that very short notes are not likely to be informative. That is, “Please see John Smith” does not allow either human annotators or an automated algorithm to predict the specialist accurately. For this reason, we did not train on or classify any notes with fewer than six tokens (enough to convey “Please see John for chest pain”). Longer notes are more likely to include patient history including diseases and symptoms unrelated to the target specialty which increases the likelihood of misclassification.

We only calculate and present wait time estimates from FP referral to specialist visit for specialists who have a model meeting our determined threshold. That is, if the performance of the best model for a particular specialty does not pass the requirements (precision of at least 0.70 and recall of at least 0.30) then we did not attempt to estimate the wait time. This decision is based on expert feedback of what percentage of notes should be captured and how precise the measurement should be to provide a trustworthy measurement. However, we acknowledge that for different methods or different datasets other minimum thresholds may be more appropriate.

To calculate the wait times, we deploy the best ML algorithm for each specialty on all EMR referral notes from 2015 (N = 174,190), i.e., the full study cohort which includes those with multiple referral notes that were not used in the development and evaluation of the machine learning approach. For each patient, for each specialty, we measure the wait time as the number of days between the first FP referral to a specialist physician and the first consultation visit of the same specialist physician, Fig 1.

To allow for adequate recall of FP referrals made at the end of the calendar year, when measuring the wait times for referrals made in 2015, we allowed for referrals to match to consultation visits occurring in 2016 (e.g., referral in November 2015 and first specialist visit in April 2016).

These models are applied on the full set of referral notes from 2015 which had at least 6 tokens (N = 174,190). Fig 4 presents the median and 90^th percentile wait times for target specialties matching our evaluation criteria.

Fig 5 compares the estimated wait times in 2015 against the latest available manually human coded estimates [11]. As the method for labelling referral notes is different the conclusions that can be made from this comparison should be made with caution. However, having developed these automated approaches, future comparisons should not be so costly to perform and would enable timely measurements of the effects of health policy on primary care wait times.

In this section, we explore the effect of choosing different confidence thresholds for the predictions of our models. Previously, we used the confidence threshold of 0.5 on classifiers which have a performance of at least 70% PPV and 30% sensitivity. As this system is being developed for a specific application purposes in our healthcare system, the threshold was selected through expert consultation with health-system researchers from our system. However, we now explore the effects of possibly changing the confidence threshold (and thus the performance thresholds), as might be desired by other systems or researchers. For each model represented in Fig 4, we plot how different confidence thresholds affect predictive performance (PPV and sensitivity) and estimated wait-times Fig 6. We observe that the confidence threshold of 0.5 (PPV> = 70% and sensitivity> = 30%) often coincides with a “stabilization” or decrease in slope for estimated wait times (e.g., clear examples can be seen in cardiology and dermatology among others). We observe that the median wait time does not differ substantially when measured either using only notes in our gold-standard set or using machine prediction to capture additional patients. However, depending on the use-case, classification might still be justified to increase the number of patients composing a class (and thus decreasing the 95% confidence interval) to allow for meaningful significance testing between different time periods.

While using automated algorithms enables classification of more referral notes, the low recall scores (0.30–0.50) make our results susceptible to skews caused by non-random mis-predictions by the algorithm. That is, although our automated method enabled us to label, on average, 167% more notes for each specialty when compared to the previous study of Jaakkimainen et al. [11], manual labelling is needed to determine if the captured notes are truly representative of the entire sample.

Furthermore, our training set was relatively small thus limiting the performance we could achieve. Our experiments demonstrate that having more notes often results in increased performance.

Another limitation is the lack of human-curated labels for referral notes. To address this limitation without incurring the high costs of human labelling, we devised a method to accurately infer the labels from administrative data: we extracted labels for patients who had only one referral and one consultation visit in 2015. However, this method limits our training set size and subsequently resulted in us not being able to train classifiers for specialties such as orthopedics where many of the referrals used custom forms thereby reducing our ability to automatically capture such labels. Furthermore, unlike a manually labelled random sample, work is needed to demonstrate that idiosyncrasies of the predictive algorithm don’t change the population being labelled.

The content of referral notes also vary amongst FPs and clinics. Busy FPs may write a referral using personalized abbreviations or few words (fewer tokens) per referral note thus making it more difficult to predict. The use of custom-stamps or forms which used tick-boxes in their referrals may also mask some of the words that could be used to label a referral note. While cross-validation would capture much of these differences on our specific dataset, more work is needed to ensure the generalizability and validity of predictions on larger and external datasets that have not been labelled (e.g., notes from later years or notes from FPs which are not in our training set).

We find an increase in the median and 90^th percentile of wait times for most specialties when compared to 2008 data. However, a direct comparison cannot be made due to the differences in referral note labelling (i.e., automated vs. human labelling). While a direct comparison cannot be made, the result from such a comparison matches those from Canadian-based surveys similarly found increased wait times in seeing specialist physicians [4, 41].