PLoS Biol plos plosbiol PLOS Biology 1544-9173 1545-7885 Public Library of Science San Francisco, CA USA 10.1371/journal.pbio.3000547 PBIOLOGY-D-19-02225 Short Reports Biology and life sciencesMicrobiologyMicrobial controlAntimicrobial resistanceAntibiotic resistance Medicine and health sciencesPharmacologyAntimicrobial resistanceAntibiotic resistance Biology and life sciencesMicrobiologyMicrobial controlAntimicrobial resistance Medicine and health sciencesPharmacologyAntimicrobial resistance Biology and life sciencesGeneticsHeredityGenetic mappingVariant genotypes Biology and life sciencesMicrobiologyMedical microbiologyMicrobial pathogensBacterial pathogensNeisseria gonorrhoeae Medicine and health sciencesPathology and laboratory medicinePathogensMicrobial pathogensBacterial pathogensNeisseria gonorrhoeae Biology and life sciencesOrganismsBacteriaNeisseriaNeisseria gonorrhoeae Medicine and health sciencesPharmacologyDrugsAntimicrobialsAntibioticsPenicillin Biology and life sciencesMicrobiologyMicrobial controlAntimicrobialsAntibioticsPenicillin Biology and life sciencesBiochemistryProteinsDNA-binding proteins Medicine and health sciencesEpidemiologyDisease surveillance Medicine and health sciencesPharmacologyDrugsAntimicrobialsAntibiotics Biology and life sciencesMicrobiologyMicrobial controlAntimicrobialsAntibiotics Surveillance to maintain the sensitivity of genotype-based antibiotic resistance diagnostics Surveillance for genotype-based AMR diagnostics http://orcid.org/0000-0003-1372-1301 Hicks Allison L. Conceptualization Data curation Formal analysis Writing – original draft Writing – review & editing 1 http://orcid.org/0000-0001-6000-8387 Kissler Stephen M. Conceptualization Formal analysis Writing – review & editing 1 http://orcid.org/0000-0003-1504-9213 Lipsitch Marc Conceptualization Formal analysis Supervision Writing – review & editing 1 2 ‡ http://orcid.org/0000-0001-5646-1314 Grad Yonatan H. Conceptualization Formal analysis Funding acquisition Supervision Writing – review & editing 1 3 ‡ * Department of Immunology and Infectious Diseases, Harvard T. H. Chan School of Public Health, Boston, Massachusetts, United States of America Center for Communicable Disease Dynamics, Department of Epidemiology, Harvard T. H. Chan School of Public Health, Boston, Massachusetts, United States of America Division of Infectious Diseases, Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, Massachusetts, United States of America Holt Kathryn Elizabeth Academic Editor University of Melbourne, AUSTRALIA

The authors have declared that no competing interests exist.

‡ These authors are co-senior authors on this work.

 * E-mail: ygrad@hsph.harvard.edu 12 11 2019 11 2019 17 11 e3000547 31 7 2019 29 10 2019 2019 Hicks et al This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

The sensitivity of genotype-based diagnostics that predict antimicrobial susceptibility is limited by the extent to which they detect genes and alleles that lead to resistance. As novel resistance variants are expected to emerge, such sensitivity is expected to decline unless the new variants are detected and incorporated into the diagnostic. Here, we present a mathematical framework to define how many diagnostic failures may be expected under varying surveillance regimes and thus quantify the surveillance needed to maintain the sensitivity of genotype-based diagnostics.

A simple mathematical framework that defines the rates of sampling and phenotypic testing necessary to efficiently detect novel resistance variants and thus maintain the sensitivity of genotype-based antimicrobial resistance diagnostics.

 National Institute of General Medical Sciences (US) U54GM088558 http://orcid.org/0000-0003-1504-9213 Lipsitch Marc http://dx.doi.org/10.13039/100000060 National Institute of Allergy and Infectious Diseases R01AI132606 http://orcid.org/0000-0001-5646-1314 Grad Yonatan H. This work was supported by Grant U54GM088558 (Models of Infectious Disease Agent Study, Center for Communicable Disease Dynamics) from the National Institute of General Medical Sciences (ML) and Grant R01AI132606 from the National Institute of Allergy and Infectious Diseases (ALH, SMK, YHG). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. PLOS Publication Stage vor-update-to-uncorrected-proof Publication Update 2019-11-22 Data Availability All data are publicly available in SRA/ENA (accession numbers provided in Table 1) or in referenced publications. Introduction

Antimicrobial resistance (AMR) poses a grave threat to global public health, underscoring the need for strategies to slow and control the spread of resistance. One direction is to develop fast and reliable diagnostics that minimize the delay between diagnosis and selection of an appropriate treatment regimen based on the target pathogen’s antibiotic susceptibility profile [1,2]. A promising approach, use of pathogen genotype to predict AMR phenotype, has been facilitated by advances in rapid and cost-efficient amplification and sequencing. For example, the Cepheid GeneXpert MTB/RIF assay for rifampicin resistance in Mycobacterium tuberculosis and the SpeeDx ResistancePlus GC assay for ciprofloxacin resistance in Neisseria gonorrhoeae are already in clinical use, and many others are in the pipeline [3–5].

These genotype-based diagnostics must maintain high sensitivity to remain useful clinically. However, the emergence of novel resistance mechanisms will inevitably lead to a decline in sensitivity, perhaps exacerbated by variable prevalence of resistance determinants across populations [6]. Key to maintaining sensitivity is, therefore, sustained sampling and routine updating of the diagnostics with newly described resistance determinants. However, despite its importance for the structure of surveillance systems and, thus, for both public health agencies and diagnostics developers, the rate of sampling necessary for timely detection of novel resistance variants has been unclear.

Here, we use datasets of clinical isolates of multiple pathogens collected over 7–14 years to show that although the sensitivities of some genetic markers of resistance remain stably high, sensitivities of other markers rapidly decline because of the emergence of novel resistance variants. We present a simple mathematical framework that defines the rates of sampling and phenotypic testing necessary for early detection of novel resistance variants.

 Results Waning sensitivity of resistance markers

In the ideal scenario for a genotype-based antibiotic resistance diagnostic, phenotypic resistance is always encoded by a specific genotype—e.g., a single, stereotyped mutation or gene. To date, some combinations of bacteria and antibiotics approximately satisfy this criterion: target modification mutations in DNA gyrase subunit A gene (gyrA) maintain high sensitivity for predicting ciprofloxacin nonsusceptibility in N. gonorrhoeae and Acinetobacter baumannii (Fig 1A–1D). For other bacteria–antibiotic combinations, diagnostic genetic markers of resistance show decreased sensitivity over time, corresponding to increased incidence of previously rare or undetected resistance markers (Fig 1E–1J). The gyrA target modification mutation in Klebsiella pneumoniae isolates [7], for example, becomes a less sensitive predictor of ciprofloxacin nonsusceptibility as the incidence of isolates with acquired Qnr family pentapeptide repeat protein gene (qnr) genes (which code for target protecting proteins) increases (Fig 1E and 1F). Similarly, the emergence of the mosaic penicillin binding protein 2 gene (penA) (XXXIV) allele in N. gonorrhoeae clinical isolates [8] corresponds to decreased sensitivity of other target modification mutations for predicting penicillin nonsusceptibility (Fig 1G and 1H). Furthermore, decreased sensitivity of carbapenem-hydrolyzing class D beta-lactamase-58 gene (blaOXA-58) for predicting imipenem nonsusceptibility in A. baumannii clinical isolates from the United States military healthcare system is associated with increased incidence of other oxacillinases (Fig 1I and 1J).

10.1371/journal.pbio.3000547.g001 Emergence of novel resistance variants and their impact on sensitivities of previous variants.

Fractional incidence (A, C, E, G, and I) and sensitivity (B, D, F, H, and J) of genetic variants over time in predicting CIP NS in N. gonorrhoeae (A-B), CIP NS in A. baumannii (C-D), CIP NS in K. pneumoniae (E-F), PEN NS in N. gonorrhoeae (G-H), and IPM NS in A. baumannii (I-J). Fractional incidence is defined as the proportion of all strains from each year that have the genetic variant or the NS phenotype. Fractional incidence of different markers may not sum to 100% due to uncharacterized resistance markers or strains carrying multiple markers. Sensitivity is defined as the fraction of NS strains from each year that have the genetic variant. Specificity (true negative rate) of variants in predicting NS is not accounted for in these plots. blaOXA, carbapenem-hydrolyzing class D beta-lactamase gene; CIP, ciprofloxacin; GyrA, DNA gyrase subunit A; IPM, imipenem; ISAba1, A. baumannii insertion sequence 1; NS, nonsusceptibility; PBP2, penicillin binding protein 2; PEN, penicillin; penA, penicillin binding protein 2 gene; qnr, Qnr family pentapeptide repeat protein gene.

 Defining required sampling rate as a function of diagnostic failure threshold

Given the possible emergence of novel resistance variants, maintenance of a genotype-based AMR diagnostic requires surveillance and phenotyping of clinical specimens predicted to be susceptible, characterization of novel resistance determinants, and subsequent updating of the diagnostic. Once a resistant strain not captured by the current diagnostic test appears in the population, there is a simple relationship between the cumulative number of such cases and the probability that at least one will be detected: if f is the proportion of all genotypically susceptible cases that receive confirmatory phenotypic testing, and N is the number of variant cases, then the probability x that the new variant is detected in at least one of those cases is given by x = 1−(1−f)N. Therefore, to have a probability of at least x that the new variant will be detected by the time N cases of it have occurred, the proportion undergoing confirmatory testing must be f≥1−(1−x)1/N

Thus, to be 95% (x = 0.95) confident that a novel variant is detected by the time it has occurred in a total of 100 (N) cases, a sampling fraction (f) of approximately 0.03 is required (i.e., 3% of incident cases must be phenotypically tested) (Fig 2A). Given the 555,608 cases of gonorrhea in the US in 2017 [9], the required sampling rate for a 95% probability of detection of a novel variant by the time it occurred in 100 cases would be 16,669 cases per year, or 1,390 cases per month (i.e., f = 3% of incident cases). For surveillance programs aimed at detecting novel resistance variants that undermine the sensitivity of a genotype-based diagnostic that has already been implemented in the population, cases with isolates predicted to be resistant by the diagnostic would be excluded from the sampling population, reducing the required sampling rate.

10.1371/journal.pbio.3000547.g002 A framework for the detection of novel resistance variants.

(A) Quantification of sampling fraction (the fraction of incident cases that are phenotyped, f) required for 95% probability (x = 0.95) of detection of novel variants as a function of the total number of novel resistance cases (N) that occur prior to detection. Sampling fractions required for 95% probability of detection of a variant that has occurred in a total of 10, 100, or 1,000 cases are indicated in panel A. (B) Estimation of the total cost associated with the phenotyping required for 95% confidence in detection of a novel resistance variant by the time is has occurred in N cases, assuming an annual case incidence (I) of 500,000, a variant growth rate (r) of 0.5 or 5 per year, and a phenotyping cost (CP) of US$20 per isolate, and the total cost associated with the mean N expected treatment failures that may be attributed to that novel variant, assuming each individual treatment failure incurs a cost (CTF) of US$100,000 or US$10,000. (C) Estimation of the cumulative cost associated with the phenotyping required for 95% confidence in detection of a novel resistance variant by the time it has occurred in N cases and the total cost incurred by the N treatment failures attributed to that novel variant, assuming an annual case incidence (I) of 500,000, a variant growth rate (r) of 0.5 or 5 per year, a phenotyping cost (CP) of US$20 per isolate, and a cost incurred by each individual treatment failure (CTF) of US$100,000 or US$10,000. Sampling fractions (f) that minimize the cumulative cost are indicated in C.

By survival analysis in which the hazard function is defined as the incidence of the novel variant multiplied by the proportion of incident cases that are phenotyped (f), if the variant has a growth rate of r (i.e., is increasing in fractional incidence [or prevalence, assuming the overall case incidence remains constant] in a population at a rate r), then the time (beginning at t0, when the variant first emerged in a single case) at which there is a probability of 1−x of having detected the variant (or an x probability of having failed to detect the variant) is t=1rln(1−rln(x)fN0) where N0 is the initial population-wide incidence of the variant in cases.

Based on this model, we can estimate the cost effectiveness of surveillance for genotype–phenotype discordance. We assume surveillance phenotyping is performed on a fraction f of all incident cases I per unit time such that there is x probability of detection of each novel variant by the time t that N cases of the novel variant have occurred. If the cost of phenotyping an individual isolate is CP, then the total cost from the phenotyping effort required to detect a novel resistance variant is CPtotal=fItCP If the cost of a treatment failure is CTF, a composite of the costs from the individual clinical failure and secondary cases, the variant occurs in the mean number of expected cases, and assuming that every attempted treatment of infection caused by a pathogen with the variant results in failure, then the expected total cost of treatment failure due to a novel resistance variant is CTFtotal=CTFf

For a pathogen with a given case incidence I (e.g., 500,000 cases per year) and phenotyping cost CP (e.g., US$20 per isolate), the total cost from the phenotyping effort required to detect a novel resistance variant and the total cost from the treatment failures that may be attributed to that novel variant can be determined for a range of assumptions about variant growth rate r and cost of treatment failure CTF (Fig 2B). Similarly, the cumulative cost associated with phenotyping and treatment failure can be assessed as a function of sampling fraction in order to identify the most cost-effective sampling fraction, defined as that which minimizes the total cost of sampling and treatment failures (Fig 2C).

 Discussion

Although this sampling model is based on few assumptions and should be generalizable to any resistance variant, the practical implementation of this model requires consideration of multiple additional factors. First, although this model assumes instantaneous testing of isolates, if the phenotyping of the collected clinical isolates is batched, then the intervals between testing could lead to delays in detection. However, the delay is bounded by the selected threshold of allowed failures, the testing interval, and the growth rate of the novel variant in the population.

Second, changes in disease incidence impact the surveillance and sampling strategy. For example, gonorrhea incidence in the US increased 65% between 2008 and 2017 and 18.6% between 2016 and 2017 alone (https://www.cdc.gov/std/stats17). To maintain the same level of confidence that the novel variant will be detected by the desired time or threshold number of cases, disease incidence would need to be closely monitored and surveillance matched accordingly. Given the directly proportional relationship between case incidence and the number of isolates that must be sampled per unit time (sampling rate) to achieve a given objective, an 18.6% increase in incidence of genotypically susceptible strains must correspond to an 18.6% increase in sampling rate in order to maintain the same sampling fraction (f) and, thus, the same confidence in detection of a novel variant by the time it has appeared in a given number of cases.

Furthermore, the incidence of clinical isolates predicted to be susceptible (susceptible case incidence), rather than overall case incidence, is of primary relevance for detecting novel resistance determinants. Thus, a third issue to consider in sampling strategy is that the susceptible case incidence may be subject to more rapid changes than overall case incidence, depending on varying selective pressures for or against resistance introduced by a variety of factors, including antibiotic use and the diagnostic itself. Thus, in establishing a plan for a sampling strategy, a conservative approach would be to account for these fluctuations by calculating the necessary sampling rate as a fraction of all cases.

Relatedly, demographic and geographic heterogeneity in selective pressures, and thus in the likelihood of emergence of novel resistance variants, introduces an additional complication in selecting the populations for surveillance and sampling. Behavioral and socioeconomic factors may contribute to differential emergence of antibiotic resistance across subpopulations [10], and certain resistance mechanisms and variants may be more likely to appear in specific subpopulations or transmission networks [6]. For example, for some pathogens, settings such as oncology and critical care units within hospitals, where antibiotic use is highest and where patients who have failed prior antibiotic therapies are likely to concentrate, may provide ideal locations for surveillance. Thus, although a diverse sampling of the population may be optimal in the absence of epidemiological analysis of risk factors for emergence of resistance, the latter may facilitate more targeted sampling strategies that help to reduce delays in detection of novel variants. Similarly, although this model assumes random sampling across a population, it should be noted that this may be difficult to achieve. For example, the Centers for Disease Control and Prevention’s Gonococcal Isolate Surveillance Project currently only samples from male patients attending selected sexual health clinics, introducing demographic and geographic bias [11]. Assessment of the impact of demographic and geographic factors on detection efficiency of novel variants may help improve sampling strategies and yield a sampling scheme–tailored model with more estimates.

Delays in updating genotype-based diagnostics may also influence the rates of emergence of new variants because these diagnostics introduce selective pressure against isolates with the diagnostic targets and increased fitness for those lacking the targets [12–14]. Thus, assay adaptability is likely to be an important determinant of diagnostic sustainability. For genotype-based diagnostics that rely on testing for specific alleles, once specimens with unknown pathways to resistance have been identified, it will be important to define the genetic basis of resistance and incorporate it into the diagnostic assay. Thus, long-term support of such diagnostics will require a system for rapidly determining the genetic basis of resistance in novel resistant variants, an activity that is currently challenging for some pathogen species with less tractable genetics and in cases of multifactorial resistance mechanisms. This requirement may create an advantage for diagnostics that rely on phylogenetic similarity [15] and are agnostic to the resistance determinant, for which genetic experiments could be avoided but regularly updating the reference database will be critical for maintaining sensitivity. However, such approaches are not likely to perform well for drugs for which resistance is frequently gained and lost through de novo mutation and/or horizontal gene transfer and thus are associated with less phylogenetic signal (e.g., as with azithromycin in N. gonorrhoeae [8]).

Estimating the costs of expected treatment failures and phenotypic testing as a function of sampling fraction may be useful for identifying the most cost-effective phenotyping rate. However, although published estimates of direct healthcare costs associated with each case of a given infectious disease may serve as a proxy for the cost of treatment failure, such estimates are likely highly variable and will need to be tailored based on factors such as the type of strain (e.g., multidrug-resistant versus extensively drug-resistant M. tuberculosis) and the progression of the disease (e.g., uncomplicated gonorrhea versus progression to pelvic inflammatory disease or epididymitis) [16,17]. It will also be important to determine how to incorporate into this estimate indirect costs such as productivity loss, further transmission, or increased antibiotic resistance due to inappropriate use. Furthermore, assessing cost effectiveness requires estimating the rate at which a novel variant can be expected to spread in the population, which may be difficult to reliably predict for all novel variants. However, cost-efficient surveillance may be achieved by tailoring models based on relevant clinical and epidemiological parameters of the pathogen and evaluations of novel variant emergence patterns after implementation of the diagnostic.

This model is based on the assumption that the most efficient and reliable method for detection of novel resistance variants is routine phenotypic testing of strains predicted to be susceptible. However, identification of treatment failures represents an additional and potentially more efficient route to detection [18]. Although the cost-effectiveness framework is based on the assumption that the vast majority of treatment failures will go undetected, depending on factors such as overall case incidence, health system factors, and severity of clinical failure associated with the pathogen, identification of treatment failures may be a more practical alternative to large-scale phenotypic sampling programs. However, identification of treatment failures may be encumbered by a number of factors, including long treatment regimens and/or partial abatement of symptoms and, thus, failure to follow up. Furthermore, infections might be cleared even in the case of undetected resistance, and multidrug therapy may similarly mask novel resistance to individual drugs. For example, one of the first identified cases of infection with the N. gonorrhoeae FC428 clone (associated with ceftriaxone resistance and intermediate azithromycin resistance) in the United Kingdom was identified as negative by N. gonorrhoeae nucleic acid amplification test (NAAT) 2 weeks after treatment with ceftriaxone and azithromycin, and a second patient in this transmission network showed clinical response to treatment with ceftriaxone and azithromycin before relapse, potentially resulting in transmission to and asymptomatic carriage in her partner [19]. Thus, although continued collection of clinical outcome data is crucial to defining the relationship between phenotypic susceptibility test results and expected treatment outcome, surveillance programs designed to regularly sample a sufficient fraction of isolates in a given population, incorporating relevant epidemiological information, may represent the most reliable strategy for comprehensive detection of novel resistance variants.

 Materials and methods

See Table 1 for details of the datasets assessed. For all datasets, raw sequence data were downloaded from the NCBI Sequence Read Archive. Genomes were assembled using SPAdes v3.13 [20] with default parameters. Assembly quality was assessed using QUAST v4.3 [21], and contigs <500 bp in length and/or with <10× average coverage were excluded. Antibiotic resistance loci were identified in the assembled contigs using BLAST [22], extracted, and aligned using MUSCLE [23] to assess the fractional incidence (the proportion of all isolates from each year that have the variant) and sensitivity (the proportion of all nonsusceptible isolates from each year that have the variant) of resistance variants. Survival analysis was used to relate sampling fractions (the proportion of incident strains receiving confirmatory phenotyping) to the cumulative number of cases of the novel variant prior to detection, the time to detection of the novel variant after emergence, and the cost of phenotyping and treatment failures. The hazard function, or the rate of identifying a strain with the novel variant given that it has not yet been detected, was defined as λ(t)=frN0e(r−μ)t where f is the fraction of incident cases phenotyped, r is the growth rate of the novel variant (the rate at which the variant is increasing in fractional incidence [or prevalence, assuming the overall case incidence remains constant] in a population), N0 is the number of cases with the novel variant at the time of emergence (assumed to be 1), μ is the rate of recovery from infection with a strain with the novel variant (assumed to be ≪r, such that [r−μ]~r), and t is the time since emergence of the novel variant.

10.1371/journal.pbio.3000547.t001 Summary of datasets.
Species Datasetdescription NS phenotype(s)(associated figure and source) NCBI SRA Study ID(s)
N. gonorrhoeae Survey from nationwide (US) clinics from 2000 to 2013; male patients only; enriched for ESC and AZM resistance CIP (Fig 1A and 1B), PEN (Fig 1G and 1H) [8] ERP008891, ERP001405, ERP000144
A. baumannii Survey from clinics and hospitals within the US military healthcare system from 2000 to 2012 CIP (Fig 1C and 1D), IPM (Fig 1I and 1J) (NCBI BioSample database, BioProject PRJNA300270) SRP065910
K. pneumoniae Survey from the Houston Methodist hospital system from 2011 to 2017; enriched for β-lactam resistance CIP (Fig 1E and 1F) [7] SRP102664, SRP110988, SRP116139

Abbreviations: AZM, azithromycin; CIP, ciprofloxacin; ESC, extended spectrum cephalosporin; ID, identifier; IPM, imipenem; NCBI SRA, National Center for Biotechnology Information Sequence Read Archive; NS, nonsusceptible; PEN, penicillin

 Abbreviations AMR

antimicrobial resistance

 blaOXA

carbapenem-hydrolyzing class D beta-lactamase gene

 GyrA

DNA gyrase subunit A

 ISAba1

A. baumannii insertion sequence 1

 NAAT

nucleic acid amplification test

 PBP2

penicillin binding protein 2

 penA

penicillin binding protein 2 gene

 qnr

Qnr family pentapeptide repeat protein gene

 References Turner KM, Christensen H, Adams EJ, McAdams D, Fifer H, McDonnell A, et al. Analysis of the potential for point-of-care test to enable individualised treatment of infections caused by antimicrobial-resistant and susceptible strains of Neisseria gonorrhoeae: a modelling study. BMJ Open. 2017;7(6):e015447. doi: 10.1136/bmjopen-2016-015447 28615273; PubMed Central PMCID: PMC5734280. McAdams D, Waldetoft KW, Tedijanto C, Lipsitch M, Brown SP. Resistance diagnostics as a public health tool to combat antibiotic resistance: A model-based evaluation. PLoS Biol. 2019;17(5):e3000250. doi: 10.1371/journal.pbio.3000250 31095567 Lo SW, Kumar N, Wheeler NE. Breaking the code of antibiotic resistance. Nat Rev Microbiol. 2018;16(5):262. doi: 10.1038/nrmicro.2018.33 29576619. Nguyen M, Long SW, McDermott PF, Olsen RJ, Olson R, Stevens RL, et al. Using machine learning to predict antimicrobial minimum inhibitory concentrations and associated genomic features for nontyphoidal Salmonella. J Clin Microbiol. 2018. doi: 10.1101/380782 Zumla A, Al-Tawfiq JA, Enne VI, Kidd M, Drosten C, Breuer J, et al. Rapid point of care diagnostic tests for viral and bacterial respiratory tract infections—needs, advances, and future prospects. Lancet Infect Dis. 2014;14(11):1123–35. doi: 10.1016/S1473-3099(14)70827-8 25189349. Andre E, Goeminne L, Colmant A, Beckert P, Niemann S, Delmee M. Novel rapid PCR for the detection of Ile491Phe rpoB mutation of Mycobacterium tuberculosis, a rifampicin-resistance-conferring mutation undetected by commercial assays. Clin Microbiol Infect. 2017;23(4):267.e5–e7. doi: 10.1016/j.cmi.2016.12.009 27998822. Nguyen M, Brettin T, Long SW, Musser JM, Olsen RJ, Olson R, et al. Developing an in silico minimum inhibitory concentration panel test for Klebsiella pneumoniae. Sci Rep. 2018;8(1):421. doi: 10.1038/s41598-017-18972-w 29323230; PubMed Central PMCID: PMC5765115. Grad YH, Harris SR, Kirkcaldy RD, Green AG, Marks DS, Bentley SD, et al. Genomic Epidemiology of Gonococcal Resistance to Extended-Spectrum Cephalosporins, Macrolides, and Fluoroquinolones in the United States, 2000–2013. J Infect Dis. 2016;214(10):1579–87. doi: 10.1093/infdis/jiw420 27638945; PubMed Central PMCID: PMC5091375. Centers for Disease Control and Prevention. Sexually Transmitted Disease Surveillance 2017. Atlanta, GA: Centers for Disease Control and Prevention; 2017 [cited 2019 Apr 20]. Available from: https://www.cdc.gov/std/gisp/gisp-protocol-feb-2015_v3.pdf. Collignon P, Beggs JJ, Walsh TR, Gandra S, Laxminarayan R. Anthropological and socioeconomic factors contributing to global antimicrobial resistance: a univariate and multivariable analysis. Lancet Planet Health. 2018;2(9):e398–e405. doi: 10.1016/S2542-5196(18)30186-4 30177008. Centers for Disease Control and Prevention. Gonococcal Isolate Surveillance Project (GISP) protocol 2016. Atlanta, GA: Centers for Disease Control and Prevention; 2016 [cited 2019 May 15]. Available from: https://www.cdc.gov/std/gisp/gisp-protocol-feb-2015_v3.pdf. Herrmann B, Torner A, Low N, Klint M, Nilsson A, Velicko I, et al. Emergence and spread of Chlamydia trachomatis variant, Sweden. Emerg Infect Dis. 2008;14(9):1462–5. doi: 10.3201/eid1409.080153 18760021; PubMed Central PMCID: PMC2603114. Smid J, Althaus CL, Low N, Unemo M, Herrmann B. The rise and fall of the new variant of Chlamydia trachomatis in Sweden: mathematical modelling study. bioRxiv [Preprint]. 2019. Available from: https://doi.org/10.1101/572107. Berhane A, Anderson K, Mihreteab S, Gresty K, Rogier E, Mohamed S, et al. Major Threat to Malaria Control Programs by Plasmodium falciparum Lacking Histidine-Rich Protein 2, Eritrea. Emerg Infect Dis. 2018;24(3):462–70. doi: 10.3201/eid2403.171723 29460730; PubMed Central PMCID: PMC5823352. Břinda K, Callendrello A, Cowley L, Charalampous T, Lee RS, MacFadden DR, et al. Lineage calling can identify antibiotic resistant clones within minutes. bioRxiv [Preprint]. 2018. Available from: https://doi.org/10.1101/403204. Chesson HW, Collins D, Koski K. Formulas for estimating the costs averted by sexually transmitted infection (STI) prevention programs in the United States. Cost Eff Resour Alloc. 2008;6:10. doi: 10.1186/1478-7547-6-10 18500996; PubMed Central PMCID: PMC2426671. Loveday M, Wallengren K, Reddy T, Besada D, Brust JCM, Voce A, et al. MDR-TB patients in KwaZulu-Natal, South Africa: Cost-effectiveness of 5 models of care. PLoS ONE. 2018;13(4):e0196003. doi: 10.1371/journal.pone.0196003 29668748; PubMed Central PMCID: PMC5906004. Berenger BM, Demczuk W, Gratrix J, Pabbaraju K, Smyczek P, Martin I. Genetic Characterization and Enhanced Surveillance of Ceftriaxone-Resistant Neisseria gonorrhoeae Strain, Alberta, Canada, 2018. Emerg Infect Dis. 2019;25(9):1660–7. doi: 10.3201/eid2509.190407 31407661; PubMed Central PMCID: PMC6711210. Eyre DW, Town K, Street T, Barker L, Sanderson N, Cole MJ, et al. Detection in the United Kingdom of the Neisseria gonorrhoeae FC428 clone, with ceftriaxone resistance and intermediate resistance to azithromycin, October to December 2018. Euro Surveill. 2019;24(10). doi: 10.2807/1560-7917.ES.2019.24.10.1900147 30862336; PubMed Central PMCID: PMC6415501. Bankevich A, Nurk S, Antipov D, Gurevich AA, Dvorkin M, Kulikov AS, et al. SPAdes: a new genome assembly algorithm and its applications to single-cell sequencing. J Comput Biol. 2012;19(5):455–77. doi: 10.1089/cmb.2012.0021 22506599; PubMed Central PMCID: PMC3342519. Gurevich A, Saveliev V, Vyahhi N, Tesler G. QUAST: quality assessment tool for genome assemblies. Bioinformatics. 2013;29(8):1072–5. doi: 10.1093/bioinformatics/btt086 23422339; PubMed Central PMCID: PMC3624806. Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ. Basic local alignment search tool. J Mol Biol. 1990;215(3):403–10. doi: 10.1016/S0022-2836(05)80360-2 2231712. Edgar RC. MUSCLE: multiple sequence alignment with high accuracy and high throughput. Nucleic Acids Res. 2004;32(5):1792–7. doi: 10.1093/nar/gkh340 15034147; PubMed Central PMCID: PMC390337. 10.1371/journal.pbio.3000547.r001 Decision Letter 0 Richardson Lauren A Senior Editor 2019 Lauren A Richardson This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 0

2 Aug 2019

Dear Dr Grad,

Thank you for submitting your manuscript entitled "Surveillance to maintain the sensitivity of genotype-based antibiotic resistance diagnostics" for consideration as a Short Reports by PLOS Biology.

Your manuscript has now been evaluated by the PLOS Biology editorial staff as well as by an academic editor with relevant expertise and I am writing to let you know that we would like to send your submission out for external peer review.

However, before we can send your manuscript to reviewers, we need you to complete your submission by providing the metadata that is required for full assessment. To this end, please login to Editorial Manager where you will find the paper in the 'Submissions Needing Revisions' folder on your homepage. Please click 'Revise Submission' from the Action Links and complete all additional questions in the submission questionnaire.

*Please be aware that, due to the voluntary nature of our reviewers and academic editors, manuscripts may be subject to delays during the holiday season. Thank you for your patience.*

Please re-submit your manuscript within two working days, i.e. by Aug 04 2019 11:59PM.

Login to Editorial Manager here: https://www.editorialmanager.com/pbiology

During resubmission, you will be invited to opt-in to posting your pre-review manuscript as a bioRxiv preprint. Visit http://journals.plos.org/plosbiology/s/preprints for full details. If you consent to posting your current manuscript as a preprint, please upload a single Preprint PDF when you re-submit.

Once your full submission is complete, your paper will undergo a series of checks in preparation for peer review. Once your manuscript has passed all checks it will be sent out for review.

Feel free to email us at plosbiology@plos.org if you have any queries relating to your submission.

Kind regards,

Lauren A Richardson, Ph.D

Senior Editor

PLOS Biology

 10.1371/journal.pbio.3000547.r002 Decision Letter 1 Richardson Lauren A Senior Editor 2019 Lauren A Richardson This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 1

3 Sep 2019

Dear Dr Grad,

Thank you very much for submitting your manuscript "Surveillance to maintain the sensitivity of genotype-based antibiotic resistance diagnostics" for consideration as a Short Reports at PLOS Biology. Your manuscript has been evaluated by the PLOS Biology editors, an Academic Editor with relevant expertise, and by several independent reviewers.

In light of the reviews (below), we are pleased to offer you the opportunity to address the comments from the reviewers in a revised version that we anticipate should not take you very long. We will then assess your revised manuscript and your response to the reviewers' comments and we may consult the reviewers again.

Of particular note, we encourage you to present this study in a manner more accessible to medical and public health professionals. Rev #3 requests a discussion of resistance mechanisms that may not be captured by this model and Rev #4 questions how the spatial distribution of mutations may impact detection.

Your revisions should address the specific points made by each reviewer. Please submit a file detailing your responses to the editorial requests and a point-by-point response to all of the reviewers' comments that indicates the changes you have made to the manuscript. In addition to a clean copy of the manuscript, please upload a 'track-changes' version of your manuscript that specifies the edits made. This should be uploaded as a "Related" file type. You should also cite any additional relevant literature that has been published since the original submission and mention any additional citations in your response.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

Before you revise your manuscript, please review the following PLOS policy and formatting requirements checklist PDF: http://journals.plos.org/plosbiology/s/file?id=9411/plos-biology-formatting-checklist.pdf. It is helpful if you format your revision according to our requirements - should your paper subsequently be accepted, this will save time at the acceptance stage.

Please note that as a condition of publication PLOS' data policy (http://journals.plos.org/plosbiology/s/data-availability) requires that you make available all data used to draw the conclusions arrived at in your manuscript. If you have not already done so, you must include any data used in your manuscript either in appropriate repositories, within the body of the manuscript, or as supporting information (N.B. this includes any numerical values that were used to generate graphs, histograms etc.). For an example see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

For manuscripts submitted on or after 1st July 2019, we require the original, uncropped and minimally adjusted images supporting all blot and gel results reported in an article's figures or Supporting Information files. We will require these files before a manuscript can be accepted so please prepare them now, if you have not already uploaded them. Please carefully read our guidelines for how to prepare and upload this data: https://journals.plos.org/plosbiology/s/figures#loc-blot-and-gel-reporting-requirements.

Upon resubmission, the editors assess your revision and assuming the editors and Academic Editor feel that the revised manuscript remains appropriate for the journal, we may send the manuscript for re-review. We aim to consult the same Academic Editor and reviewers for revised manuscripts but may consult others if needed.

We expect to receive your revised manuscript within one month. Please email us (plosbiology@plos.org) to discuss this if you have any questions or concerns, or would like to request an extension. At this stage, your manuscript remains formally under active consideration at our journal; please notify us by email if you do not wish to submit a revision and instead wish to pursue publication elsewhere, so that we may end consideration of the manuscript at PLOS Biology.

When you are ready to submit a revised version of your manuscript, please go to https://www.editorialmanager.com/pbiology/ and log in as an Author. Click the link labelled 'Submissions Needing Revision' where you will find your submission record.

Thank you again for your submission to our journal. We hope that our editorial process has been constructive thus far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Lauren A Richardson, Ph.D

Senior Editor

PLOS Biology

*****************************************************

Reviews

Reviewer #1:

Antibiotic resistance evolution is a significant problem and one way that is currently pursued to tackle the problem is the development of rapid diagnostics. These allow to not only identify the infectious agent, but more importantly the resistance profile and often mutation. A fundamental problem with this, as basically with almost all current approaches to tackle resistance, is that evolutionary change is an ongoing process and this fact is ignored. The paper presented here provides a tool to address this issue: it shows a way to estimate how much testing is required to detect newly evolved resistant variants, given the limits of current tests. As such, this is an interesting and worthwhile approach.

Overall, the paper is well written. I find though, that it lacks in detail and could also be clearer, given that the message is partly addressed at medical professionals with little time to read and digest.

Lines 48-49. It is no really clear where these data are coming from. In the section on lines 58-73, two publications and NCBI accessions are cited, but it is mostly not clear to which panel in figure 1 they refer. Please clarify the source of the data for each panel of figure 1.

Figure 1: I think it would be easier to read if fractional incidence and sensitivity were presented on separate panels. This would, especially in panels D and E make it much easier to see the differences between the dashed lines, which contain the main information. Also, I would suggest to briefly explain fractional incidence and sensitivity in the figure legend, to allow readers to understand the figure without going back to the text.

Lines 82 – 97. The approach is nice and simple. Yet, to reach a wider readership, it could be better explained. First, I would suggest making figure 1F a separate figure with a more informative caption. Also, I would suggest to explain, given the equation, how you arrived at the figures in lines 88 and following, as some readers will not spend much time on the equations.

Another aspect worth considering here would be that, as also shown in figure 1, different variants might emerge. A brief discussion whether or not that matters would be useful.

---------------

Reviewer #2:

In this study, the author has filled an important research gap by presenting a mathematical framework to define the sampling rates for confirmatory phenotypic testing so as to detect novel or previously uncommon resistance genotypes. By updating genotype-based diagnostics, the sensitivity of genotype-based antibiotic resistance should therefore maintained high. In addition, the authors also discussed multiple factors that require consideration when using the sampling model. Overall, it is well-structured and written.

Please briefly describe where the datasets were collected from, hospital or other settings, country?

Line 61-63 As the cumulative sensitivity remains close to 1 in figure 1C and 1D, I am confused about "the genetic markers of resistance show decreased sensitivity over time". I guess the authors meant “For others, the original diagnostic genetic markers of resistance show ….”

Line 100 why the probability of having detected is 1 - X? when x was defined as the probability of new variant will be detected by time N in line 83-84.

Line 101 Please provide details on how this equation was derived. Is it possible to provide an example like 90-92?

Line 115 How was 3 and 15 additional cases calculated? Was it based on the equation in line 101.

Figure 1A As the fractional incidence in GyrA S91F is a little bit higher than CIP NS in 2000, wondering why the sensitivity for GyrAS91F is 0.

Figure 1F Is X = 0.95?

---------------

Reviewer #3:

This is a very concise and thoughtful communication on some fundamental aspects of a new (and emerging) method for susceptibility testing. Although the theory is clear, I wonder whether real-life practice might not be (much) more complicated. Resistance can be based on a single genetic event (mutation or allele or gene) as in most of the examples put forward, but also on the combination of multiple events, such as for beta-lactam resistance in Enterobacteriales. For instance the single presence of OXA48 in Klebsiella may still render a susceptible phenotype for imipenem, but addition of any other beta-lactamase may render a non-susceptible phenotype (see Dautzenberg et al, Euro Surveillance 2014 Mar 6;19(9)). Not sure how this would influence the surveillance scheme (and if the authors could elaborate on this).

---------------

Reviewer #4:

Nicholas G. Davies, signed review

Review of: Surveillance to maintain the sensitivity of genotype-based antibiotic resistance diagnostics

In this manuscript, the authors address an important question for managing antibiotic resistance: how much phenotypic testing for antibiotic resistance is needed to maintain the sensitivity of genotype-based assays for antibiotic resistance?

This is an interesting question with direct implications for policy. The manuscript is accompanied with well-chosen examples illustrating the problem of declining sensitivity of genotype-based diagnostics. There is also a good discussion of the context for the research and of considerations for putting suggestions into practice, as well as of alternative ways to maintain the sensitivity of diagnostics besides surveillance.

At the same time, the main result (line 87) is relatively straightforward to derive, which I think justifies a request that the authors go into a little more detail. For me, there is a slight disconnect here between the practical nature of the problem that is being addressed and the way in which the results are presented.

Specifically, I think the results could be rephrased (or elaborated) to be more relevant to policymakers. While it is interesting to know the required rate of testing, f, such that a new variant is detected with 100x per cent confidence by the time N variant cases have occurred (line 87), policymakers might be more interested in knowing the rate of testing f that maximizes the cost-effectiveness of surveillance, given the cost of testing, the cost of diagnostic failure, the sensitivity of the phenotypic assay for resistance (which may not be 100%, for example if there is a mixed infection), and so on.

Similarly, I’m not sure the time before detection of a novel variant (line 101) is as interesting to policymakers as the expected number of diagnostic failures before detection. Also, as presented, this result depends upon a growth rate r which is probably quite difficult to predict from first principles—after all, we are talking about the relative fitness of novel mutations—and which would no longer be needed by the time it could be measured. Conversely, the number of failures before detection would not depend on r (assuming instantaneous testing).

I have made specific suggestions here but am open to alternatives—just suggesting more generally that the paper would be improved if the results were more directly translatable to decision-making.

Another potential issue that the manuscript doesn’t seem to address is that the model assumes that isolates subjected to phenotypic testing are selected randomly with respect to the overall population being monitored. But the model risks being overconfident if, for example, the relative rate of phenotypic testing varies spatially, since novel mutations will not in general be spread evenly through a population.

Minor issues:

Lines 90-92: The meaning is clear, but there should be a statement about 95% confidence in this sentence.

Lines 98-102: It’s not quite clear from the way this is phrased whether a variant which is remaining at a stable frequency over time should have r = 0 or r = 1. Also, the probability of having detected the variant, which used to be x, now seems to be 1 – x, which is a little confusing. Finally, slightly more detail on how line 101 was derived would be clarifying.

Line 175: the meaning of “identified as NAAT-negative” is a bit opaque—can this be rephrased?

Sincerely,

Nick Davies

London School of Hygiene and Tropical Medicine

 10.1371/journal.pbio.3000547.r003 Author response to Decision Letter 1 Submission Version 2

17 Sep 2019

Submitted filename: phenotypic_sampling_response_to_reviewers_9-16.docx

 10.1371/journal.pbio.3000547.r004 Decision Letter 2 Richardson Lauren A Senior Editor 2019 Lauren A Richardson This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 2

16 Oct 2019

Dear Dr Grad,

Thank you for submitting your revised Short Reports entitled "Surveillance to maintain the sensitivity of genotype-based antibiotic resistance diagnostics" for publication in PLOS Biology. I have now obtained advice from three of the original reviewers and have discussed their comments with the Academic Editor.

As you will read, the reviewers all found your work very well revised. Based on the reviews, we will probably accept this manuscript for publication, assuming that you will modify the manuscript to meet our remaining production requirements. Of note, the manuscript needs and Methods and Materials section.

We expect to receive your revised manuscript within two weeks. Before we will be able to formally accept your manuscript and consider it "in press", we also need to ensure that your article conforms to our guidelines. A member of our team will be in touch shortly with a set of requests. As we can't proceed until these requirements are met, your swift response will help prevent delays to publication.

Upon acceptance of your article, your final files will be copyedited and typeset into the final PDF. While you will have an opportunity to review these files as proofs, PLOS will only permit corrections to spelling or significant scientific errors. Therefore, please take this final revision time to assess and make any remaining major changes to your manuscript.

Please note that you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

Please note that an uncorrected proof of your manuscript will be published online ahead of the final version, unless you opted out when submitting your manuscript. If, for any reason, you do not want an earlier version of your manuscript published online, uncheck the box. Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us as soon as possible if you or your institution is planning to press release the article.

To submit your revision, please go to https://www.editorialmanager.com/pbiology/ and log in as an Author. Click the link labelled 'Submissions Needing Revision' to find your submission record. Your revised submission must include a cover letter, a Response to Reviewers file that provides a detailed response to the reviewers' comments (if applicable), and a track-changes file indicating any changes that you have made to the manuscript.

Please do not hesitate to contact me should you have any questions.

Sincerely,

Lauren A Richardson, Ph.D

Senior Editor

PLOS Biology

------------------------------------------------------------------------

DATA POLICY:

You may be aware of the PLOS Data Policy, which requires that all data be made available without restriction: http://journals.plos.org/plosbiology/s/data-availability. For more information, please also see this editorial: http://dx.doi.org/10.1371/journal.pbio.1001797

Note that we do not require all raw data. Rather, we ask that all individual quantitative observations that underlie the data summarized in the figures and results of your paper be made available in one of the following forms:

1) Supplementary files (e.g., excel). Please ensure that all data files are uploaded as 'Supporting Information' and are invariably referred to (in the manuscript, figure legends, and the Description field when uploading your files) using the following format verbatim: S1 Data, S2 Data, etc. Multiple panels of a single or even several figures can be included as multiple sheets in one excel file that is saved using exactly the following convention: S1_Data.xlsx (using an underscore).

2) Deposition in a publicly available repository. Please also provide the accession code or a reviewer link so that we may view your data before publication.

**Please ensure that figure legends in your manuscript and the Data Statement in the submission system include information on where the underlying data can be found.

------------------------------------------------------------------------

BLOT AND GEL REPORTING REQUIREMENTS:

For manuscripts submitted on or after 1st July 2019, we require the original, uncropped and minimally adjusted images supporting all blot and gel results reported in an article's figures or Supporting Information files. We will require these files before a manuscript can be accepted so please prepare them now, if you have not already uploaded them. Please carefully read our guidelines for how to prepare and upload this data: https://journals.plos.org/plosbiology/s/figures#loc-blot-and-gel-reporting-requirements.

------------------------------------------------------------------------

Reviews

Reviewer #2:

The manuscript has greatly improved and the authors have addressed the comments/questions that I raised. Thank you.

--------------

Reviewer #3: Yes: Marc Bonten

No further comments

--------------

Reviewer #4: Yes: Nicholas G. Davies

The authors have satisfactorily addressed my concerns. This paper would make a good contribution to PLOS Biology.

 10.1371/journal.pbio.3000547.r005 Author response to Decision Letter 2 Submission Version 3

24 Oct 2019

Submitted filename: phenotypic_sampling_response_to_reviewers_9-16.docx

 10.1371/journal.pbio.3000547.r006 Decision Letter 3 Richardson Lauren A Senior Editor 2019 Lauren A Richardson This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 3

29 Oct 2019

Dear Dr Grad,

On behalf of my colleagues and the Academic Editor, Kathryn Elizabeth Holt, I am pleased to inform you that we will be delighted to publish your Short Reports in PLOS Biology.

The files will now enter our production system. You will receive a copyedited version of the manuscript, along with your figures for a final review. You will be given two business days to review and approve the copyedit. Then, within a week, you will receive a PDF proof of your typeset article. You will have two days to review the PDF and make any final corrections. If there is a chance that you'll be unavailable during the copy editing/proof review period, please provide us with contact details of one of the other authors whom you nominate to handle these stages on your behalf. This will ensure that any requested corrections reach the production department in time for publication.

Early Version

The version of your manuscript submitted at the copyedit stage will be posted online ahead of the final proof version, unless you have already opted out of the process. The date of the early version will be your article's publication date. The final article will be published to the same URL, and all versions of the paper will be accessible to readers.

PRESS

We frequently collaborate with press offices. If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximise its impact. If the press office is planning to promote your findings, we would be grateful if they could coordinate with biologypress@plos.org. If you have not yet opted out of the early version process, we ask that you notify us immediately of any press plans so that we may do so on your behalf.

We also ask that you take this opportunity to read our Embargo Policy regarding the discussion, promotion and media coverage of work that is yet to be published by PLOS. As your manuscript is not yet published, it is bound by the conditions of our Embargo Policy. Please be aware that this policy is in place both to ensure that any press coverage of your article is fully substantiated and to provide a direct link between such coverage and the published work. For full details of our Embargo Policy, please visit http://www.plos.org/about/media-inquiries/embargo-policy/.

Thank you again for submitting your manuscript to PLOS Biology and for your support of Open Access publishing. Please do not hesitate to contact me if I can provide any assistance during the production process.

Kind regards,

Hannah Harwood

Publication Assistant,

PLOS Biology

on behalf of

Lauren Richardson,

Senior Editor

PLOS Biology