Discussion
In this study the lower bound for the specificity of the SD Biosensor Standard Q AG test was 96.4%. This worst case analysis provides only a lower bound of the true specificity since it neglects entirely the true incidence of COVID-19. As the antigen tests in Slovakia were not controlled directly using accompanying PCR-tests, we refrain from an attempt to subdivide the observed positives into true and false positives.
The above estimate is consistent with the information provided by the manufacturer [14], stating 0.32% (0.01, 1.78) as a false positive rate—a very broad CI. The derived upper bound from the mass testing is more informative and appears relevant in that it constitutes the first (to the best of our knowledge) bound obtained from large scale practical use of Standard Q and also suggests better performance of Standard Q compared to previous studies: a large study (with 2347 SARS-CoV-2-free samples based on a PCR-test), suggests a specificity of 99.3% (CI 98.6–99.6) [7]. Other available specificity estimates are based on an order of magnitude smaller samples (99.2% (CI 97.1–99.8) [4] and 100% [6] and, in addition, are from study populations with high incidence rates (according to PCR-testing). A point estimate of 98.53% is given by [5], based on 100 SARS-Cov2-free samples with other respiratory viruses present and 35 samples from healthy volunteers.
Data from Phase 2 was used in this study, because the general countrywide testing was performed irrespective of regional incidence rates, covering also regions with potentially very low incidence. The data has also been used to investigate spatial patterns of the spread of COVID-19 [10] and to demonstrate that both mass-scale testing and restrictive measures contributed to the sharp drop in COVID-19 incidence [9]. However, to avoid a positively biased specificity estimate, compared to [9] we use an updated, more detailed dataset, where an erroneous positivity rate for Bratislava IV has been corrected. The binomial distribution assumption underlying the computation of the Clopper-Pearson confidence intervals, is based on the assumption of independent events that could be violated if, e.g. testing stations were operating with different quality. However, the Bonferroni-correction is a strictly conservative approach to derive simultaneous confidence bounds that account for the selection of the county with the lowest upper bound to derive the estimate. It is well understood in the epidemiology literature that imperfect reference criteria lead to a systematic bias of the true incidence [15]. Specifically, the sensitivity and specificity of tests are systematically underestimated if the criterion (SARS-CoV-2 infection) cannot be assessed directly and tests are compared to a surrogate criterion as gold standard which is subject to errors. This could be a contributing factor for the relatively higher specificity found in Slovak data compared to other studies. However, there are other conceivable factors that could bias the results: different quality of swab sampling and handling, low temperature in outdoor testing stations, deviations in the production over time and data quality.
While the analysis is based on a worst case assumption, considering all positives to be false positives, the obtained estimate appears relevant in that it constitutes the first bound obtained from large scale practical use of Standard Q. It suggests better specificity of Standard Q compared to previous studies and these findings can support the planning and justification of future mass testing programs. Besides test specificity, the sensitivity is essential for the evaluation of the overall utility of mass testing strategies. Unfortunately, based on the mass testing data set no estimate of the test sensitivity can be obtained and, to our knowledge, no sensitivity estimates have been provided in a mass-testing setting so far such that further studies are needed.
10.1371/journal.pone.0255267.r001
Decision Letter 0
Ito
Etsuro
Academic Editor
2021
Etsuro Ito
This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Submission Version
0
26 May 2021
PONE-D-21-14542
Analysis of the specificity of a COVID-19 antigen test in the Slovak mass testing program
PLOS ONE
Dear Dr. Posch,
Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.
I obtained the comment from one reviewer but could not obtain it from the other reviewer. To progress the smooth review process, I have determined to move to the next round for the review.
Please submit your revised manuscript by Jul 08 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.
Please include the following items when submitting your revised manuscript:
A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.
If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.
If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.
We look forward to receiving your revised manuscript.
Kind regards,
Etsuro Ito
Academic Editor
PLOS ONE
Journal Requirements:
When submitting your revision, we need you to address these additional requirements.
1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at
https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and
https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf
2. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.
3. We note you have included a table to which you do not refer in the text of your manuscript. Please ensure that you refer to Table 1 in your text; if accepted, production will need this reference to link the reader to the Table.
Reviewers' comments:
Reviewer's Responses to Questions
Comments to the Author
1. Is the manuscript technically sound, and do the data support the conclusions?
The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.
Reviewer #1: Yes
**********
2. Has the statistical analysis been performed appropriately and rigorously?
Reviewer #1: Yes
**********
3. Have the authors made all data underlying the findings in their manuscript fully available?
The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.
Reviewer #1: Yes
**********
4. Is the manuscript presented in an intelligible fashion and written in standard English?
PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.
Reviewer #1: Yes
**********
5. Review Comments to the Author
Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)
Reviewer #1: In presenting argument in the introduction that antibody testing can be an alternative for RT-CPR, it may be helpful to note that the test evaluates different things: active infection vs history of infection (including current), with different delays between tests being positive (longer for serology). This reviewer agrees that serology can be a very attractive alternative for reasons cited by the authors, but additional caveats other than SN (sensitivity) and SP (specificity) related to meaning of the test in terms of inference of onset of incident infection should be clearly presented.
Can you please be clearer in the introduction how your approach is “in contrast to other studies which use a PCR-test as a reference to estimate sensitivity and specificity of an Ag-Test4”?
One alternative to Bonferroni correction that seems appealing to this reviewer is to model rate of positive tests using binomial regression with random effect of county, and maybe some fixed effects that can account for rate of positive tests, such as numbers quarantined per county, positivity rates from Phase 1, age structure, economic indicators, percent of county tested, outdoor temperature, count/proportion of outdoor test sites, date of test (the usual confounders in epidemiology that can affect both willingness to test and chance of having been infected plus those mentioned in the discussion by the authors). The fixed intercept of the model would overall rate of positive tests and can be made county-specific through use of other random and fixed covariates; upper percentile of these modelled estimates can be used authors did with their Bonferroni-adjusted confidence interval. One may be also tempted to model spatial correlation, if descriptive statistics support its existence.
Instead of just giving upper bound of 95%CI, it would be more informative to give both the estimate per county and associated standard error. This should allow anyone to estimate different percentile if for their purposes 97.5% is not appropriate. This may already be in the supplemental materials, so maybe this is just a matter of pointing the reader in the right direction.
**********
6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.
If you choose “no”, your identity will remain anonymous but your review may still be made public.
Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.
Reviewer #1: Yes: Igor Burstyn
While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.