Milk samples from dairy cows with clinical mastitis arriving at the Mastitis Laboratory, at the Swedish Veterinary Agency (SVA) in Uppsala, Sweden, were used in this study. SVA is a reference laboratory and the Mastitis Laboratory is accredited according to ISO 17025. At the Mastitis Laboratory milk samples were handled in parallel and analysed with two different methods. The milk samples used in this study came from two different sources, either dairy farms with a Bacticam station in use (Agricam clients) or other farms. Approval to use the milk samples for research purpose was obtained from the sample owner, either directly (by signing a form) or indirectly (by accepting the terms and conditions when sending a sample to SVA). Ethical approval for the study was not warranted according to the Swedish animal welfare ordinance (2019:66). Milk samples arriving at the Mastitis Laboratory from the 8^th of July 2021 to the 6^th of April 2022 were included in the study. The distribution of milk samples over time and sample origin are presented in Supporting information (S1 Fig).

At the Mastitis Laboratory, milk samples were processed according to NMC recommendations [14]. In brief, 10 µl of milk were cultured aerobically on a blood agar plate at 37° C and evaluated after 24 and 48 hours of growth. At evaluation, number of colonies were registered and initially characterised based on morphology and haemolysis. For bacteriological identification, bacterial colonies from agar plates were typed to species level through MALDI-Tof analysis. If necessary, an extended bacteriological examination was performed, in which case the detection limit was lowered by spreading 100 µl of milk on the agar plate. Bacterial diagnosis at the Mastitis Laboratory is based on colony morphology and a sample is considered positive if at least two CFU of a major pathogen or at least five CFU of a minor pathogen are identified. If growth of two different colony types are detected the sample is classified as having a mixed flora and if three or more different colonies are detected, the sample is considered contaminated. However, if a highly contagious (major) pathogen is identified among numerous different colonies the sample will be classified as positive for both, i.e., co-infected.

In parallel with the accredited method, personnel from the Mastitis Laboratory spread 4 x 10 µl of milk from each sample on a SELMA + agar plate and photographed the growth in a Bacticam photo studio provided by Agricam. The Bacticam photo studio is a light impervious cubical box approximately 20 x 20 x 20 cm large, with a slot for the agar plate, a camera holder and internal illumination. Apart from the photo studio, equipment for growing bacteria and a smartphone camera with Internet access is required for the Bacticam bacterial analysis. After spreading, SELMA + agar plates were incubated aerobically at 37 °C and photos were taken after 24 and 48 hours of growth. Images from the photographed SELMA + agar plates were temporarily stored in a OneDrive folder before being downloaded to a SLU server to which only the authors had access.

Images of the photographed SELMA + agar plates were analysed in an off-line image classifier system that was provided by Agricam to the authors. The image classifier system is a two-step process where images first are quality controlled and then classified by a pre-trained image classifier algorithm. The demands for the off-line image classifier algorithm were that 4 images were available, these 4 images needed to be taken with illumination of the agar plate from above and below with a minimum of 18 hours apart. Samples with less than 4 images available were excluded. Samples with more than 4 images available were manually curated so that surplus images were excluded from the analysis. The bacterial classifier can only identify one bacterial diagnosis per sample. For the commercial on-line Bacticam system, the company have routines for correcting surplus or missing images, the option of a preliminary bacteriological diagnosis after 8–24 hours as well as a backup system with manual inspection or laboratory analysis for un-diagnosed samples.

Results from the (gold) standard laboratory culture and the off-line image classifier are the primary source of data for this study. A third source of data was provided by the company Agricam and consisted of image classifier results from on-farm Bacticam stations from milk samples that were also sent to the Mastitis Laboratory during the period 8^th of July to 29^th of December 2021.

Raw data was compiled using Microsoft Excel and data analysis was performed in the statistics program R [15]. Analysis was performed on three levels.

First, an overview of how the classifier performs in practice was performed (general performance). This includes descriptive presentation (e.g., to which extent bacterial diagnoses are provided), exploration of patterns for when a diagnosis was not provided and how the classifier performed for samples with co-infection. Identified bacterial species were grouped to the lowest comparable taxonomic level. Bacteriological species (/genera) for which the classifier had no pre-training were categorized as “Other” and the classifier was expected to classify them as “Additional evaluation required”. These patterns were explored descriptively.

In the second level, diagnoses provided by the classifier were compared to diagnoses from the reference (gold standard) laboratory to calculate sensitivity, specificity and predictive values (specific performance). In this comparison only samples for which the bacterial classifier returned a diagnosis and where the reference laboratory specified one bacterial diagnosis per sample were included (i.e., samples with co-infection and samples with a specified species in mixed flora were excluded). Identified bacterial species were grouped to the lowest comparable taxonomic level. For calculation of sensitivity, specificity and predictive values the package DTCompair was used [16] and for visualization the package ggplot2 was used [17].

The third level of the analysis includes comparisons of results from the on-line classifier from on-farm Bacticam stations (third source of data) in relation to the results of the off-line classifier at the reference laboratory and the reference laboratory results. Descriptively exploring overall accordance, the effect of sample transportation and sample handling.

During the study period, 1299 milk samples were analysed by the Mastitis Laboratory, the majority of the samples arrived between September and December, 52% (672/1299) of the milk samples came from farms with a Bacticam station in use and 48% of the samples came from other farms (Supl. S1 Fig ). All samples were cultured and photographed in the photo studio. Images from 87 samples could not be analysed by the classifier due to poor image quality or fewer than 4 images per sample and were excluded from the study. Of the remaining 1212 samples, 1077 samples had one specific bacterial diagnose identified per sample and 135 samples had multiple pathogens identified per sample. Extended bacteriological examination was performed by the Mastitis Laboratory on 110 samples. The classifier returned bacterial diagnoses for 845 samples (70%) while 367 (30%) required additional evaluation. A summary of the classifiers’ response in relation to the Mastitis Laboratory diagnosis is presented in Table 1 and as a more detailed list in Supporting information (S1 Table).

Of the 367 samples that according to the classifier required additional evaluation, 261 samples had bacterial growth that the image classifier theoretically could identify. 57 samples had growth of multiple bacterial species in the same sample and 49 samples had growth of a single bacterial species for which the classifier had not been trained, i.e., for 106 samples the correct Bacticam diagnosis would be “additional evaluation required” (Table 1). Further, 32 out of the 367 samples had been subjected to extended bacteriological examination, the majority (27/32) of these samples were classified as “no growth” by the Mastitis Laboratory, in the remaining five samples, either E. coli or Staphylococci in mixed flora were identified.

A comparison of Mastitis Laboratory and classifier results for samples with one bacterial diagnosis per sample is presented in Fig 2. The most frequently occurring bacterial diagnoses were E. coli, Streptococcus spp., mixed flora and S. aureus. The fifth most common bacterial diagnosis was “other species”, i.e., species for which the classifier had not been trained.

In total, 110 samples had, according to the Mastitis Laboratory, growth of species for which the classifier had not been trained. Forty-nine of these 110 samples received a correct “additional evaluation required” diagnosis by the classifier while an incorrect bacterial diagnosis was returned for 61 samples. The most commonly returned diagnosis in these cases was Strep. spp. (n = 18), mixed flora (n = 14) and negative growth (n = 14) (Fig 2).

For calculation of sensitivity, specificity and predictive values, 767 samples for which the Mastitis Laboratory had provided one single bacterial diagnosis per sample and the bacterial classifier had provided a bacterial diagnosis were used (Table 2). The most frequently occurring pathogens according to the Mastitis Laboratory were E. coli (n = 169), S. aureus (n = 112) and non-beta-haemolytic streptococci (n = 176). Samples with no growth and samples with mixed flora were also relatively common (n = 66 and n = 99) while NAS, Klebsiella and beta haemolytic streptococci were less common (n < 35). The sensitivity and specificity of the bacterial classifier varied between bacterial species, with the best performance for the more frequently occurring pathogens (Fig 3). The specificity of the bacterial classifier was high for all bacterial diagnoses with a range from 91.9% to 99.1%. For the sensitivity there was a larger variation and the confidence intervals were broader. The most frequently occurring pathogens (E. coli, S. aureus and non-beta haemolytic streptococci) had relatively high sensitivity (> 85%) while the sensitivity for the less common pathogens (Klebsiella spp., NAS and β-haemolytic streptococci was lower (43.5%–64.3%) (Fig 3). A table with estimated mean values for sensitivity, specificity, PPV and NPV together with standard error and confidence interval is presented in Supporting information (S2 Table).

To evaluate the effect of handling and transportation, bacterial classifier results from the on-line on-farm Bacticam stations provided by Agricam were used. 130 milk samples had comparable results from the Mastitis Laboratory, the off-line bacterial classifier and the on-line bacterial classifier. For these 130 samples, 144 bacterial diagnoses were provided by the Mastitis Laboratory, 117 milk samples that had one bacterial diagnosis per sample were further explored. 72 (62%) of the 117 samples had the same bacteriological diagnosis given by the on-line classifier, the off-line classifier and the reference laboratory, and 69% of the samples (81/117) had the same bacteriological diagnosis given by the on-line classifier and the off-line bacterial classifier. There was a higher concordance of results between the Mastitis Laboratory and the off-line bacterial classifier (Bacticam at laboratory, 89 shared diagnoses) than between the Mastitis Laboratory and the on-line classifier (Bacticam on farm, 78 shared diagnoses). A presentation of the 117 samples that had one bacterial diagnosis per sample and comparable results from the on-line classifier, the off-line bacterial classifier and the Mastitis Laboratory is presented in Fig 4. An interesting observation is that 26% of the samples that were cultured on farm and evaluated by the on-line image classifier had growth of mixed flora, when the same samples were cultured at the reference laboratory and evaluated by the off-line bacterial classifier, 11% of the samples had growth of mixed flora. As the sample results provided by Agricam contained detailed information on when samples were collected, the time from sampling to arrival at the reference laboratory was calculated. The mean transportation time for the milk sample was 4.9 days with a range from 1 – 26 days.

Sensitivity and specificity were evaluated on samples where the Mastitis Laboratory and the bacterial classifier had provided a single bacterial diagnosis per sample. The specificity was overall high for all bacterial diagnoses provided by the classifier, implying that samples without a specified diagnosis really are free from it. A high specificity is important to assure that animals don’t receive unnecessary treatments. For the sensitivity there was a larger variation in results. For the most common bacterial diagnoses in this data set (E. coli, S. aureus, non-haemolytic streptococci and no growth) the sensitivity was high, indicating that if the classifier give this response it is most likely correct. For less common bacterial diagnoses (Klebsiella spp., NAS, beta-haemolytic streptococci and mixed flora) the sensitivity was lower and the results less trustworthy. The distribution of bacterial diagnoses in the data set that was provided as training data for the neural network is unknown to us authors, it is not unlikely that the training data had a similar distribution of diagnoses as presented here and that the classifier therefore is better trained on certain diagnoses. For example, E. coli and Klebsiella spp. have similar but distinct growth patterns on a SELMA + agar plate, with a colour change in the PGUA agar as differentiating factor, this colour change is visually apparent and we do find it probable that the colour change would be possible to identify for an automated image classifier, consequently we find it likely that the sensitivity for Klebsiella spp. will increase with more training. The specificity values reported for the Bacticam bacterial classifier in this study (91.9-99.1%) are in line with specificity values reported for a similar application, “Rumi” (96-99%), by Garcia et al. [13]. A variation in sensitivity across different bacterial diagnoses that is noted here was likewise noted by Garcia et al.

It is well known that storage can affect the hygienic properties and bacterial content of raw milk. According to literature, total bacterial count will increase with prolonged storage, especially if milk is stored over 4 ^°C [24]. For microorganisms typically found in milk, it has been shown that there is a large variability in growth rate, both within and between different organisms [25]. This difference in growth rate would affect the original composition of bacteria in a milk sample, and the difference would increase with time. This has been shown in a master degree thesis that concluded that mixed bacterial flora increased with increased number of days of transportation [26]. In the present study, the average transportation time was 4.9 days, the conditions during transport are unknown but milk samples sent to the Mastitis Laboratory are most often transported at ambient temperature (personal message). Considering the wide range in transportation time (1 - 26 days), we cannot exclude that some samples were stored before transportation. Regardless of transportation temperature, the time spent for transportation to the laboratory prevent the analysis from being used as a decision tool for selective treatment. Considering the transportation time, we did expect that there would be more samples with mixed bacterial growth after transportation compared to before, surprisingly the results were opposite (26% mixed infection before transport vs. 11% after). We can only speculate on why this difference occurred, on the farm either the farmer or a trained employee would be responsible for spreading the milk on the agar while in the lab a professional lab-assistant did it. It is possible that the difference in “professionalism” contributed to the difference in results regarding mixed infection and concordance between analysing methods. However, it cannot be ruled out that handling and storage of agar plates or other factors contributed.