PLOS Digit Health plos plosdh PLOS Digital Health 2767-3170 Public Library of Science San Francisco, CA USA PDIG-D-23-00131 10.1371/journal.pdig.0000324 Review Physical sciencesMathematicsApplied mathematicsAlgorithms Research and analysis methodsSimulation and modelingAlgorithms Biology and life sciencesAnatomyCardiovascular anatomyHeart Medicine and health sciencesAnatomyCardiovascular anatomyHeart Biology and life sciencesPhysiologyCardiovascular physiology Medicine and health sciencesDiagnostic medicineAuscultation Physical sciencesMathematicsApplied mathematicsAlgorithmsMachine learning algorithms Research and analysis methodsSimulation and modelingAlgorithmsMachine learning algorithms Computer and information sciencesArtificial intelligenceMachine learningMachine learning algorithms Medicine and health sciencesWomen's healthMaternal healthPregnancy Medicine and health sciencesWomen's healthObstetrics and gynecologyPregnancy Medicine and health sciencesPediatrics Medicine and health sciencesDiagnostic medicine Heart murmur detection from phonocardiogram recordings: The George B. Moody PhysioNet Challenge 2022 PhysioNet Challenge 2022 https://orcid.org/0000-0003-4688-7965 Reyna Matthew A. Conceptualization Data curation Formal analysis Funding acquisition Investigation Methodology Project administration Resources Software Supervision Validation Visualization Writing – original draft Writing – review & editing 1 * Kiarashi Yashar Formal analysis Investigation Methodology Writing – original draft Writing – review & editing 1 Elola Andoni Conceptualization Data curation Formal analysis Methodology Software Visualization Writing – original draft Writing – review & editing 2 Oliveira Jorge Conceptualization Data curation Methodology Resources Writing – original draft Writing – review & editing 3 Renna Francesco Conceptualization Data curation Formal analysis Methodology Resources Writing – original draft Writing – review & editing 4 Gu Annie Resources Software 1 Perez Alday Erick A. Resources 1 Sadr Nadi Resources 1 5 Sharma Ashish Resources 1 Kpodonu Jacques Writing – review & editing 6 Mattos Sandra Conceptualization Data curation Funding acquisition Investigation Project administration Resources Supervision 7 https://orcid.org/0000-0001-7501-6523 Coimbra Miguel T. Conceptualization Data curation Funding acquisition Investigation Project administration Resources Supervision 4 https://orcid.org/0000-0003-4913-6825 Sameni Reza Conceptualization Data curation Formal analysis Funding acquisition Investigation Methodology Project administration Resources Software Supervision Validation Visualization Writing – original draft Writing – review & editing 1 Rad Ali Bahrami Conceptualization Data curation Formal analysis Investigation Methodology Project administration Resources Software Supervision Writing – original draft Writing – review & editing 1 Clifford Gari D. Conceptualization Data curation Formal analysis Funding acquisition Investigation Methodology Project administration Resources Software Supervision Validation Visualization Writing – original draft Writing – review & editing 1 8 Department of Biomedical Informatics, Emory University, Atlanta, Georgia, United States of America Department of Electronic Technology, University of the Basque Country UPV/EHU, Eibar, Spain REMIT, Universidade Portucalense, Porto, Portugal INESC TEC, Faculdade de Ciências, Universidade do Porto, Porto, Portugal ResMed, Sydney, Australia Division of Cardiac Surgery, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, United States of America Unidade de Cardiologia e Medicina Fetal, Real Hospital Português, Recife, Brazil Department of Biomedical Engineering, Emory University and the Georgia Institute of Technology, Atlanta, Georgia, United States of America Celi Leo Anthony Editor Massachusetts Institute of Technology, UNITED STATES

I have read the journal’s policy and the authors of this manuscript have the following competing interests: AE receives financial support through grant PID2021-122727OB-I00 funded by MCIN/AEI/10.13039/501100011033 and "ERDF A way of making Europe" and by the Basque Government under Grant IT1717-22. FR and MC receive financial support by National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia, within project UIDB/50014/2020. GC has financial interests in AliveCor, LifeBell AI and Mindchild Medical. GC also holds a board position in LifeBell AI and Mindchild Medical.

¤ Current address: Department of Biomedical Informatics, Emory University, Atlanta, Georgia, United States

 * E-mail: matthew.a.reyna@emory.edu 9 2023 11 9 2023 2 9 e0000324 7 4 2023 3 7 2023 2023 Reyna et al This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Cardiac auscultation is an accessible diagnostic screening tool that can help to identify patients with heart murmurs, who may need follow-up diagnostic screening and treatment for abnormal cardiac function. However, experts are needed to interpret the heart sounds, limiting the accessibility of cardiac auscultation in resource-constrained environments. Therefore, the George B. Moody PhysioNet Challenge 2022 invited teams to develop algorithmic approaches for detecting heart murmurs and abnormal cardiac function from phonocardiogram (PCG) recordings of heart sounds. For the Challenge, we sourced 5272 PCG recordings from 1452 primarily pediatric patients in rural Brazil, and we invited teams to implement diagnostic screening algorithms for detecting heart murmurs and abnormal cardiac function from the recordings. We required the participants to submit the complete training and inference code for their algorithms, improving the transparency, reproducibility, and utility of their work. We also devised an evaluation metric that considered the costs of screening, diagnosis, misdiagnosis, and treatment, allowing us to investigate the benefits of algorithmic diagnostic screening and facilitate the development of more clinically relevant algorithms. We received 779 algorithms from 87 teams during the Challenge, resulting in 53 working codebases for detecting heart murmurs and abnormal cardiac function from PCG recordings. These algorithms represent a diversity of approaches from both academia and industry, including methods that use more traditional machine learning techniques with engineered clinical and statistical features as well as methods that rely primarily on deep learning models to discover informative features. The use of heart sound recordings for identifying heart murmurs and abnormal cardiac function allowed us to explore the potential of algorithmic approaches for providing more accessible diagnostic screening in resource-constrained environments. The submission of working, open-source algorithms and the use of novel evaluation metrics supported the reproducibility, generalizability, and clinical relevance of the research from the Challenge.

 Author summary

Cardiac auscultation is an accessible diagnostic screening tool for identifying heart murmurs. However, experts are needed to interpret heart sounds, limiting the accessibility of auscultation. Therefore, the George B. Moody PhysioNet Challenge 2022 invited teams to develop algorithms for detecting heart murmurs and abnormal cardiac function from phonocardiogram (PCG) recordings of heart sounds. For the Challenge, we sourced 5272 PCG recordings from 1452 primarily pediatric patients in rural Brazil. We required the participants to submit the complete code for their algorithms, improving the transparency, reproducibility, and utility of their work. We also devised an evaluation metric that considered the costs of screening, diagnosis, misdiagnosis, and treatment, allowing us to investigate the benefits of algorithmic diagnostic screening and facilitate the development of more clinically relevant algorithms. We received 779 algorithms from 87 teams during the Challenge, resulting in 53 working codebases and publications that represent a diversity of algorithmic approaches to detecting heart murmurs and identifying clinical outcomes from heart sound recordings.

 http://dx.doi.org/10.13039/100000057 National Institute of General Medical Sciences 2R01GM104987-09 Clifford Gari D. http://dx.doi.org/10.13039/100000070 National Institute of Biomedical Imaging and Bioengineering R01EB030362 Clifford Gari D. http://dx.doi.org/10.13039/100006108 National Center for Advancing Translational Sciences UL1TR002378 Clifford Gari D. http://dx.doi.org/10.13039/100000936 Gordon and Betty Moore Foundation Clifford Gari D. http://dx.doi.org/10.13039/100014600 MathWorks Clifford Gari D. ERDF A way of making Europe IT1717-22 Elola Andoni http://dx.doi.org/10.13039/501100005856 Faculdade de Ciências e Tecnologia, Universidade Nova de Lisboa UIDB/50014/2020 Oliveira Jorge http://dx.doi.org/10.13039/501100005856 Faculdade de Ciências e Tecnologia, Universidade Nova de Lisboa UIDB/50014/2020 Renna Francesco http://dx.doi.org/10.13039/100014600 MathWorks https://orcid.org/0000-0003-4688-7965 Reyna Matthew A. http://dx.doi.org/10.13039/100000936 Gordon and Betty Moore Foundation https://orcid.org/0000-0003-4688-7965 Reyna Matthew A. http://dx.doi.org/10.13039/100000057 National Institute of General Medical Sciences 2R01GM104987-09 https://orcid.org/0000-0003-4688-7965 Reyna Matthew A. http://dx.doi.org/10.13039/100000070 National Institute of Biomedical Imaging and Bioengineering R01EB030362 https://orcid.org/0000-0003-4688-7965 Reyna Matthew A. http://dx.doi.org/10.13039/100006108 National Center for Advancing Translational Sciences UL1TR002378 https://orcid.org/0000-0003-4688-7965 Reyna Matthew A. This research is supported by the National Institute of General Medical Sciences (NIGMS) and the National Institute of Biomedical Imaging and Bioengineering (NIBIB) under NIH grant numbers 2R01GM104987-09 and R01EB030362 respectively, the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR002378, as well as the Gordon and Betty Moore Foundation and MathWorks under unrestricted gifts. AE receives financial support through grant PID2021-122727OB-I00 funded by MCIN/AEI/10.13039/501100011033 and "ERDF A way of making Europe" and by the Basque Government under Grant IT1717-22. FR and MC receive financial support by National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia, within project UIDB/50014/2020. None of the aforementioned entities influenced the design of the Challenge or provided data for the Challenge. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data Availability J. H. Oliveira, F. Renna, P. Costa, D. Nogueira, C. Oliveira, C. Ferreira, A. Jorge, S. Mattos, T. Hatem, T. Tavares, A. Elola, A. Rad, R. Sameni, G. D. Clifford, & M. T. Coimbra (2021). The CirCor DigiScope Dataset: From Murmur Detection to Murmur Classification. IEEE Journal of Biomedical and Health Informatics, https://doi.org/10.1109/JBHI.2021.3137048. The Challenge data can be downloaded from Oliveira, J., Renna, F., Costa, P., Nogueira, M., Oliveira, A. C., Elola, A., Ferreira, C., Jorge, A., Bahrami Rad, A., Reyna, M., Sameni, R., Clifford, G., & Coimbra, M. (2022). The CirCor DigiScope Phonocardiogram Dataset (version 1.0.3). PhysioNet. https://doi.org/10.13026/tshs-mw03. Introduction

Cardiac auscultation via stethoscopes remains the most common and the most cost-effective tool for cardiac pre-screening. However, despite its popularity, cardiac auscultation has limited diagnostic sensitivity and accuracy because the interpretation of heart sounds from stethoscopes requires many years of experience and skill, leading to significant disagreement between medical personnel [1, 2, 3, 4]. Digital phonocardiography has emerged as a more objective alternative to traditional cardiac auscultation. The phonocardiogram (PCG) can be acquired by a combination of high-fidelity stethoscope front-ends and high-resolution digital sampling circuitry, which enable the registration of heart sounds as a discrete-time signal and enable the use of algorithmic methods for heart sound analysis and diagnosis [5].

As acoustic signals, heart sounds are mainly generated by the vibrations of cardiac valves as they open and close during the cardiac cycle. Turbulent blood flow may cause vibrations around the cardiac valves that create audible heart sounds and abnormal waveforms in the PCG, which are called murmurs. Different kinds of murmurs exist, and they are characterized in various ways, including location, timing, duration, shape, intensity, and pitch. The identification and analysis of murmurs can provide valuable information about cardiovascular pathologies.

However, while cardiac auscultation itself is relatively accessible, experts are needed to interpret heart sounds and PCGs, limiting the accessibility of auscultation for cardiac disease management, especially in resource-constrained environments, where limited access to clinical experts and infrastructure prevents widespread screening. The ability to correctly interpret PCGs for detecting murmurs and identifying different pathologies requires time and broad clinical experience. Therefore, the objective interpretation of the PCG remains a difficult skill to acquire.

The 2022 George B. Moody PhysioNet Challenge (formerly the PhysioNet/Computing in Cardiology Challenge) provided an opportunity to address these issues by inviting teams to develop automated approaches for detecting heart murmurs and abnormal cardiac function from PCG recordings. We sourced and shared PCGs from a largely pediatric population in Brazil, and we asked teams to use the PCGs to identify both heart murmurs and the clinical outcomes of a full diagnostic screening. The Challenge explored the potential of algorithmic approaches for interpreting heart sound recordings as part of diagnostic screening for more accessible cardiac care.

 Methods Challenge data

The CirCor DigiScope dataset [6] was used for the George B. Moody PhysioNet Challenge 2022. This dataset consists of 5268 PCG recordings from one or more auscultation locations during 1568 patient encounters with 1452 distinct patients. The patient population was primarily pediatric, ranging from neonates to adolescents, but it also included pregnant adults; no recordings were collected from fetuses. The dataset was collected during two screening campaigns in Paraíba, Brazil from July 2014 to August 2014 and from June 2015 to July 2015. The study protocol was approved by the 5192-Complexo Hospitalar HUOC/PROCAPE Institutional Review Board under the request of the Real Hospital Português de Beneficiência em Pernambuco. A detailed description of the dataset can be found in [6].

During the data collection sessions, each participant answered a sociodemographic questionnaire, followed by a clinical examination (anamnesis and physical examination), a nursing assessment (physiological measurements), and cardiac investigations (cardiac auscultation, chest radiography, electrocardiogram, and echocardiogram). The collected data were then analyzed by an expert pediatric cardiologist. The expert could re-auscultate the participant and/or request complementary tests. At the end of a session, the pediatric cardiologist either directed the participant for a follow-up appointment, referred the participant to cardiac catheterization or heart surgery, or discharged the participant.

The PCGs were recorded using an electronic auscultation device, the Littmann 3200 stethoscope, from up to four auscultation locations on the body; see Fig 1:

Aortic valve: second intercostal space, right sternal border;

Pulmonic valve: second intercostal space, left sternal border;

Tricuspid valve: lower left sternal border; and

Mitral valve: fifth intercostal space, midclavicular line (cardiac apex).

 10.1371/journal.pdig.0000324.g001 Auscultation locations for the CirCor DigiScope dataset [<xref ref-type="bibr" rid="pdig.0000324.ref006">6</xref>], which was used for the Challenge: Pulmonary valve (PV), aortic valve (AV), mitral valve (MV), and tricuspid valve (TV).

The choice of locations, the number of recordings at each location, and the duration of the PCGs varied between patients. The PCGs were recorded by multiple operators, but the PCGs for each patient encounter were recorded by a single operator, and they were recorded in a sequential manner, i.e., not simultaneously. The PCGs were also inspected for signal quality and semi-automatically segmented using the three algorithms proposed in [7], [8], and [9] and then corrected, as deemed necessary, by a cardiac physiologist.

Each patient’s PCGs and clinical notes were also annotated for murmurs and abnormal cardiac function (described below). These annotations served as the labels for the Challenge.

The murmur annotations and characteristics (location, timing, shape, pitch, quality, and grade) were manually identified by a single cardiac physiologist independently of the available clinical notes and PCG segmentation. The cardiac physiologist annotated the PCGs by listening to the audio recordings and by visually inspecting the corresponding waveforms. The murmur annotations indicated whether the annotator could detect the presence or absence of a murmur in a patient from the PCG recordings for the patient or whether the annotator was unsure about the presence or absence of a murmur. The murmur annotations did not indicate whether a murmur was pathological or innocent.

The clinical outcome annotations were determined by cardiac physiologists using all available clinical notes, including the socio-demographic questionnaire, clinical examination, nursing assessment, and cardiac investigations. In particular, these notes include reports from an echocardiogram, which is a standard diagnostic tool for characterizing cardiac function. The clinical outcome annotations indicated whether the expert annotator identified normal or abnormal cardiac function. The clinical outcome annotations were performed by different experts, and these experts were different from the expert who performed the murmur annotations.

In total, the Challenge dataset consisted of 5272 annotated PCG recordings from 1568 patient encounters with 1452 patients. We released 60% of the recordings in a public training set and retained 10% of the recordings in a private validation set and 30% of the recordings in a private test set. The training, validation, and test sets were matched to approximately preserve the univariate distributions of the variables in the data. Data from patients who participated in multiple screening campaigns belonged to only one of the training, validation, or test sets to prevent data leakage. We shared the training set at the beginning of the Challenge to allow the participants to develop their algorithms, and we sequestered the validation and test sets during the Challenge to evaluate the submitted the algorithms.

Table 1 summarizes the variables provided in the training, validation, and test sets of the Challenge data. Table 2 summarizes the distributions of the variables provided with the data.

10.1371/journal.pdig.0000324.t001 Demographic, murmur, and clinical outcome information in the Challenge training, validation, and/or test sets; <monospace specific-use="no-wrap">nan</monospace> values indicate unknown or missing values.
Variable Description Possible values Dataset splits
Age Reported age Neonate (birth to 27 days), Infant (28 days to 1 year), Child (1 to 11 years), Adolescent (12 to 18 years), nan (unknown) Training, validation, test
Sex Reported sex Female, Male Training, validation, test
Height Height in centimeters Positive number or nan (unknown) Training, validation, test
Weight Weight in kilograms Positive number or nan (unknown) Training, validation, test
Pregnancy status Reported pregnancy status True, False Training, validation, test
Murmur Indicates if a murmur is present, absent, or unidentifiable or unknown for the annotator; a Challenge label Present, Absent, Unknown Training
Murmur locations Auscultation locations for observed murmurs PV, TV, AV, MV, Phc, nan (unknown); concatenated with + Training
Most audible location Auscultation location where murmurs sounded most intense PV, TV, AV, MV, Phc, nan (unknown) Training
Systolic murmur timing Timing of the murmur within the systolic period Early-systolic, Mid-systolic, Late-systolic, Holosystolic, nan (unknown) Training
Systolic murmur shape Shape of the murmur in the systolic period Crescendo, Decrescendo, Diamond, Plateau, nan (unknown) Training
Systolic murmur pitch Pitch of the murmur in the systolic period Low, Medium, High, nan (unknown) Training
Systolic murmur grading Grading of the murmur in the systolic period according to a modified Levine scale [10] I/VI, II/VI, III/VI, nan (unknown) Training
Systolic murmur quality Quality of the murmur in the systolic period Blowing, Harsh, Musical, nan (unknown) Training
Diastolic murmur timing Timing of the murmur within the diastolic period Early-diastolic, Mid-diastolic, Holodiastolic, nan (unknown) Training
Diastolic murmur shape Shape of the murmur in the diastolic period Decrescendo, Plateau, nan (unknown) Training
Diastolic murmur pitch Pitch of the murmur in the diastolic period Low, Medium, High, nan (unknown) Training
Diastolic murmur grading Grading of the murmur in the diastolic period I/IV, II/IV, III/IV, nan (unknown) Training
Diastolic murmur quality Quality of the murmur in the diastolic period Blowing, Harsh, nan (unknown) Training
Outcome Indicates a normal or abnormal clinical outcome as diagnosed by the medical expert; a Challenge label Normal, Abnormal Training
Campaign Screening campaign attended by the patient CC2014, CC2015 Training
Additional ID Other patient identifier for patients who attended both screening campaigns Patient identifier Training
10.1371/journal.pdig.0000324.t002 Demographic, murmur, and clinical outcome distributions across the Challenge training, validation, and test data.

For categorical variables, the entries of the table denote the fraction of the dataset with each possible value. For numerical variables, the entries of the table denote the median and first and third quartiles, respectively, of the values in the dataset, i.e., median [Q1, Q3].

Variable Training set Validation set Test set Entire dataset
Age
- Neonate 0.006 0.007 0.006 0.006
- Infant 0.134 0.101 0.107 0.122
- Child 0.705 0.678 0.723 0.708
- Adolescent 0.076 0.094 0.099 0.085
- nan 0.079 0.121 0.065 0.078
Sex
- Female 0.516 0.530 0.453 0.498
- Male 0.484 0.470 0.547 0.502
Height 115 [89, 133] cm 116 [89, 136] cm 113 [92, 135] cm 115 [89, 134] cm
Weight 20.4 [12.5, 31.2] kg 20.9 [12.9, 34.6] kg 21.0 [13.4, 32.7] kg 20.6 [12.7, 32.0] kg
Pregnancy status
- False 0.926 0.899 0.948 0.930
- True 0.074 0.101 0.052 0.070
Murmur
- Absent 0.738 0.711 0.719 0.730
- Unknown 0.072 0.107 0.073 0.076
- Present 0.190 0.181 0.208 0.195
Outcome
- Normal 0.516 0.617 0.493 0.518
- Abnormal 0.484 0.383 0.507 0.482
Campaign
- CC2014 0.409 0.403 0.436 0.416
- CC2015 0.591 0.597 0.564 0.584
Challenge objective

We designed the Challenge to explore the potential for algorithmic pre-screening of heart murmurs and abnormal heart function, especially in resource-constrained environments [18]. We asked the Challenge participants to design working, open-source algorithms for detecting heart murmurs and abnormal cardiac function from PCG recordings. For each patient encounter, each algorithm interpreted the PCG recordings and/or demographic data.

Challenge timeline

The George B./ Moody PhysioNet Challenge 2022 was the 23rd George B. Moody PhysioNet Challenge [11, 18]. As with previous Challenges, the 2022 Challenge had an unofficial phase and an official phase.

The unofficial phase (February 1, 2022 to April 8, 2022) introduced the teams to the Challenge. We publicly shared the Challenge objective, training data, example algorithms, and evaluation metrics at the beginning of the unofficial phase. At this time, we only had access to the patients’ murmur annotations, so we only asked the teams to detect murmurs. We invited the teams to submit the entries with the code for their algorithms for evaluation, and we scored at most 5 entries from each team on the hidden validation set during the unofficial phase.

Between the unofficial phase and official phase, we took a hiatus (April 9, 2022 to April 30, 2022) to improve the Challenge in response to feedback from the teams, the broader community, and our collaborators. During this time, we added the patients’ clinical outcomes for abnormal cardiac function to the CirCor DigiScope dataset [6].

The official phase (May 1, 2022 to August 15, 2022) allowed the teams to refine their approaches for the Challenge. We updated the Challenge objectives, data, example algorithms, and evaluation metric at the beginning of the official phase. At this time, we had access to both the patients’ murmur annotation and clinical outcomes, so we asked the teams to detect murmurs and abnormal cardiac function. We again invited the teams to submit their entries for evaluation, and we scored at most 10 entries from each team on the hidden validation set during the official phase.

After the end of the official phase, we asked each team to choose a single entry from their team for evaluation on the test set. We allowed the teams to choose any successful model from the official phase, but most teams chose their best-scoring entries. We only evaluated one entry from each team to prevent sequential training on the test set. The winners of the Challenge were the teams with the best scores on the test set. We announced the winners at the end of the Computing in Cardiology (CinC) 2022 conference.

The teams presented and defended their work at CinC 2022, which was held in Tampere, Finland. As described in the next section, the teams wrote four-page conference proceeding paper describing their work, which we reviewed for accuracy and coherence. The code for their algorithms will be publicly released after the end of the Challenge and the publication of the papers on the Challenge website.

 Challenge rules and expectations

We encouraged the teams to ask questions, pose concerns, and discuss the Challenge in a public forum, but we prohibited them from discussing or sharing their work during the unofficial phase, hiatus, or official phase of the Challenge to preserve the diversity and uniqueness of their approaches.

For both phases of the Challenge, we required the teams to submit the complete code for their algorithms, including their preprocessing, training, and inference steps. We first ran each team’s training code on the public training set to train the models. We then ran the trained models on the hidden validation and test sets to label the data; we ran the trained models on each patient sequentially to reflect the sequential nature of the screening process. We then scored the outputs from the models using the expert annotations on hidden validation and test sets.

We allowed the teams to submit either MATLAB or Python code; other programming languages were considered upon request, but there were no requests for other programming languages. Participants containerized their code in Docker and submitted it by sharing private GitHub or Gitlab repositories with their code. We downloaded their code and ran it in containerized environments on Google Cloud. We described the computational architecture of these environments more fully in [12].

Each entry had access to 8 virtual CPUs, 52 GB RAM, 50 GB local storage, and an optional NVIDIA T4 Tensor Core GPU (driver version 470.82.01) with 16 GB VRAM. We imposed a 72-hour time limit for training each entry on the training set without a GPU, a 48-hour time limit for training each entry on the training set with a GPU, and a 24-hour time limit for running each trained entry on either the validation or test set either with or without a GPU.

To aid the teams, we shared example MATLAB and Python entries. These examples used random forest classifiers with the age group, sex, height, weight, pregnancy status of the patient as well as the presence, mean, variance, and skewness of the numerical values in each PCG recording as features. We did not design these example entries to perform well. Instead, we designed them to provide minimal working examples of how to read the Challenge data and write the model outputs.

 Challenge evaluation

To capture the focus of the 2022 Challenge on algorithmic screening for heart murmurs and abnormal cardiac function, we developed novel scoring metrics for each of the two Challenge tasks: detecting heart murmurs and identifying clinical outcomes for abnormal or normal heart function.

As described above, the murmurs were directly observable from the PCG recordings, but the clinical outcomes were determined by a more comprehensive diagnostic screening, including the interpretation of an echocardiogram. However, despite these differences, we asked the teams to perform both tasks using only the PCGs and routine demographic data, which allowed us to explore the diagnostic potential of algorithmic approaches for the interpretation of relatively easily accessible PCGs.

The algorithms for both of these tasks effectively pre-screened patients for expert referral. Under this paradigm, if an algorithm inferred potentially abnormal cardiac function, i.e., the model outputs were murmur present, murmur unknown, or outcome abnormal, then the algorithm would refer the patient to a human expert for a confirmatory diagnosis and potential treatment. If the algorithm inferred normal cardiac function, i.e., if the model outputs were murmur absent or outcome normal, then the algorithm would not refer the patient to an expert and the patient would not receive treatment. Fig 2 illustrates this algorithmic pre-screening process as part of a larger diagnostic pipeline.

10.1371/journal.pdig.0000324.g002 Screening and diagnosis pipeline for the Challenge.

All patients would receive algorithmic pre-screening, and patients with positive results from algorithmic pre-screening would receive confirmatory expert screening and diagnosis. (i) Patients with positive results from algorithmic pre-screening and expert annotators would receive treatment; they are true positive cases. Patients with positive results from algorithmic pre-screening and negative results from expert annotators would not receive treatment; they are false positive cases or false alarms. Patients with negative results from algorithmic pre-screening who would have received positive results from the expert annotators would have missed or delayed treatment; they are false negative cases. Patients with negative results from algorithmic pre-screening who would have also received negative results from expert annotators also would not receive treatment; they are true negative cases.

For the murmur detection task, we introduced a weighted accuracy metric that assessed the ability of an algorithm to reproduce the results of a skilled human annotator. For the clinical outcome identification task, we introduced a cost-based scoring metric that reflected the cost of expert diagnostic screening as well as the costs of timely, missed, and delayed treatment for abnormal cardiac function. The team with the highest weighted accuracy metric won the murmur detection task, and the team with the lowest cost-based evaluation metric won the clinical outcome identification task.

We formulated versions of both of these evaluation metrics for both tasks to allow for more direct comparisons; see S1 Appendix for the additional metrics. We also calculated several traditional evaluation metrics to provide additional context to the performance of the models.

Cost-based scoring is controversial, in part, because healthcare costs are an imperfect proxy for health needs [13, 14]; we reflect on this important issue in the Discussion section. However, screening costs necessarily limit the ability to perform screening, especially in more resource-constrained environments, so we considered costs as an imperfect proxy for improving access to cardiac screening.

 Weighted accuracy metric

We introduced a weighted accuracy metric to evaluate the murmur detection algorithms. This metric assessed the ability of the algorithms to reproduce the decisions of the expert annotator. This weighted accuracy metric is similar to the traditional accuracy metric, but it assigned more weight to patients that had or potentially had murmurs than to patients that did not have murmurs. These weights reflect the rationale that, in general, a missed diagnosis is more harmful than a false alarm.

We defined the weighted accuracy metric for the murmur detection task as a murmur = 5 m PP + 3 m UU + m AA 5 ( m PP + m UP + m AP ) + 3 ( m PU + m UU + m AU ) + ( m PA + m UA + m AA ) , where Table 3 defines a three-by-three confusion matrix M = [mij] for the murmur present, murmur unknown, and murmur absent classes.

10.1371/journal.pdig.0000324.t003 Confusion matrix <italic>M</italic> for murmur detection with three classes: Murmur present, murmur unknown, and murmur absent.

The columns are the ground truth labels from the human annotator, and the rows are the model outputs. The entries of the confusion matrix provide the number of patients with each model output for each ground truth class.

Murmur Expert
Present Unknown Absent
Murmur Classifier Present mPP mPU mPA
Unknown mUP mUU mUA
Absent mAP mAU mAA

These coefficients were chosen to reflect the trade-off between false positives and false negatives, where clinicians may tolerate multiple false alarms to avoid a single missed diagnosis. In (1), murmur present cases have five times the weight of murmur absent cases, and the murmur unknown cases have three times the weight of murmur absent cases, to reflect a tolerance of five false alarms for every one false positive.

Like the traditional accuracy metric, this metric only rewarded algorithms for correctly labeling patients, but it provided the highest reward for correctly classifying patients with murmurs and the lowest reward for correctly classifying patients without murmurs. It provided an intermediate reward for correctly classifying patients of unknown murmur status to reflect the difficulty and importance of indicating when the recordings are not adequate for diagnosis.

We used (1) to rank the Challenge algorithms for the murmur detection task. The team with the highest value of (1) won this task.

 Cost-based evaluation metric

We introduced a cost-based evaluation metric to evaluate the clinical outcome algorithms for abnormal or normal heart function. This metric considered the ability of the algorithms to reduce the costs associated with diagnosing and treating patients, primarily by having experts screen fewer patients with normal cardiac function. We again emphasize that healthcare costs are an imperfect surrogate for health needs [13, 14]. However, costs remain a practical consideration for resource-constrained environments.

For each patient encounter, the algorithm interpreted the PCG recordings and demographic data for the encounter. Under this paradigm, if an algorithm inferred abnormal cardiac function, then it would refer the patient to a human expert for a confirmatory diagnosis. If the expert confirmed the diagnosis, then the patient would receive treatment, and if the expert did not confirm the diagnosis, then the patient would not receive treatment. Alternatively, if the algorithm inferred normal cardiac function, then it would not refer the patient to an expert, and the patient would not receive treatment, even if the patient had abnormal cardiac function that would have been detected by a human expert.

We associated each of these steps with a cost: the cost of algorithmic pre-screening, the cost of expert screening, the cost of timely treatment, and the cost of delayed or missed treatment.

For simplicity, we assumed that algorithmic pre-screening had a relatively small cost that depended linearly on the number of algorithmic pre-screenings. We also assumed that both timely treatments and delayed or missed treatments had relatively large costs that, on average, depended linearly on the number of patients with delayed or missed treatment. Given our focus on the ability of algorithmic pre-screening to reduce human screening of patients with normal cardiac function, we assumed that expert screening had an intermediate cost that depended non-linearly on the number of screenings as well on as the infrastructure and capacity of the healthcare system. Of course, treatment costs are also non-linear in the number of treated patients for similar reasons, but non-urgent treatments can arguably utilize the capacity of the broader healthcare system. Screening far below the capacity of the healthcare system was inefficient and incurred a low total cost but high average cost. Screening above the capacity of the healthcare system was also inefficient and incurred both a high average cost and a high total cost.

Therefore, we introduced the following cost-based evaluation metric for identifying clinical outcomes. We defined the total cost of diagnosis and treatment with algorithmic pre-screening as c outcome total = f algorithm ( n patients ) + f expert ( n TP + n FP , n patients ) + f treatment ( n TP ) + f error ( n FN ) , where Table 4 defines a two-by-two confusion matrix N = [nij] for the classes for abnormal and normal cardiac function, npatients = nTP + nFP + nFN + nTN is the total number of patient encounters, and falgorithm, fexpert, ftreatment, ferror are defined below.

10.1371/journal.pdig.0000324.t004 Confusion matrix <italic>N</italic> for clinical outcome detection with two classes: Clinical outcome abnormal and clinical outcome normal.

The columns are the ground truth labels from the human annotator, and the rows are the classifier outputs. The entries of the confusion matrix provide the number of patients with each classifier output for each ground truth class.

Clinical Outcome Expert
Abnormal Normal
Clinical Outcome Classifier Abnormal nTP nFP
Normal nFN nTN

Again, for simplicity, we assumed that the costs for algorithmic pre-screening, timely treatments, and missed or late treatments were linear. We defined f algorithm ( s ) = 10 s as the total cost of s pre-screenings by an algorithm, f treatment ( s ) = 10000 s as the total cost of s treatments, and f error ( s ) = 50000 s as the total cost of s missed or delayed treatments.

Conversely, to better capture the potential benefits of algorithmic pre-screening, we assumed that the cost for expert screening was non-linear. We defined f expert ( s , t ) = ( 25 + 397 s t - 1718 s 2 t 2 + 11296 s 4 t 4 ) t as the total cost of s screenings by a human expert out of a population of t patient encounters so that g expert ( x ) = 25 + 397 x - 1718 x 2 + 11296 x 4 was the mean cost of screening a fraction x = s/t of the patient cohort by an expert; this reparameterization of (6) allowed us to compare algorithms on datasets with different numbers of patients. We designed (6) and (7) so that the mean cost of expert screening was lowest when only 25% of the patient cohort received an expert screening to reflect the optimal capacity of the screening system. Figs 3 and 4 show these costs across different patient cohort and screening sizes, and S2 Appendix provides a fuller derivation of (6) and (7).

10.1371/journal.pdig.0000324.g003 The expert screening cost <italic>g</italic><sub>expert</sub>(<italic>x</italic>) defined for the Challenge: Mean cost for screening a fraction <italic>x</italic> of a patient cohort for cardiac abnormalities.

Mean per-patient expert screening cost gexpert(x), i.e., the total expert screening cost for a patient cohort divided by the number of patients in the cohort.

 10.1371/journal.pdig.0000324.g004 The expert screening cost <italic>g</italic><sub>expert</sub>(<italic>x</italic>) defined for the Challenge: Mean cost for screening a fraction <italic>x</italic> of a patient cohort for cardiac abnormalities.

Mean per-screening expert screening cost gexpert(x)/x, i.e., the total expert screening cost for a patient cohort divided by the number of patients in the cohort and the fraction of expert screenings for the cohort.

To more easily compare costs across databases with different numbers of patients, e.g., the training, validation, and test sets, we defined the mean per-patient cost of diagnosis and treatment with algorithmic pre-screening as c outcome = c outcome total n patients .

We used (8) to rank the Challenge algorithms for the murmur detection task. The team with the lowest value of (8) won this task.

 Challenge results Challenge entries

We received 779 entries from 87 teams throughout the course of the 2022 PhysioNet Challenge, resulting in 77 submitted CinC abstracts, 62 accepted CinC abstracts, 43 published CinC proceedings papers, and 53 final entries with working code from 53 different teams.

These entries represent a diversity of approaches to the Challenge. A total of 81 teams submitted 293 entries during the unofficial phase of the Challenge, and a total of 63 teams submitted 486 entries during the official phase of the Challenge. Of the 779 total entries, we received 652 entries from 75 teams that were implemented in Python, and 127 entries from 17 teams that were implemented in MATLAB. We received 519 entries from 60 teams that requested a graphics processing unit (GPU) for their entries, and 260 entries from 49 teams requested only a central processing unit (CPU), i.e., no GPU. In total, we received 473 successful entries that we were able to train on the public training set and evaluate on the hidden validation set, and 306 entries that we were unable to train on the training set and/or evaluate on the validation set due to various errors in the submitted code.

We allowed each team with a successful entry during the official phase to select a single entry for evaluation on the test set. A total of 58 teams had a successful entry during the official phase that we were able to train on the training set and evaluate on the validation set. Most of these teams chose their best-scoring entry from the official phase, but some teams selected their most recent entry or another entry, often because they thought that another entry with a lower score on the validation set would better generalize to the test set.

In some cases, a team’s best-scoring entry for the murmur detection task was different from the team’s best-scoring entry for the clinical outcome identification task because of the differences between the tasks and the evaluation metrics for grading them; in these cases, we chose a different entry for each task. However, since the teams could implement different approaches for each task, and since they could have submitted these different approaches as part of the same entry, we did not distinguish between teams and entries that submitted their best scoring entries in the same code submission or in different code submissions. We will share the best scoring entries for both whether they were part of the same or different code submissions.

Of the 58 teams with a successful entry during the official phase, 53 teams had code that we were able to score on the training, validation, and test sets for both the murmur detection and clinical outcome identification tasks, resulting in 53 working entries to the Challenge.

Fig 5 compares the performance of the working Challenge entries on the murmur detection and clinical outcome identification tasks. Figs 6 and 7 compare the Challenge weighted accuracy and cost metrics, respectively, with traditional scoring metrics, including area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPRC), the macro-averaged F-measure, and the traditional accuracy metric.

10.1371/journal.pdig.0000324.g005 Weighted accuracy metric (<xref ref-type="disp-formula" rid="pdig.0000324.e001">1</xref>) for the murmur detection task (<italic>x</italic>-axis) and the cost metric (<xref ref-type="disp-formula" rid="pdig.0000324.e008">8</xref>) scores for the clinical outcome identification task (<italic>y</italic>-axis) of the final Challenge entries on the hidden test set.

Each point shows an entry, and the shading of each point shows whether the entry was an official entry (dark red points) that satisfied all of the Challenge rules or an unofficial entry (light red points) that did not. The Spearman correlation coefficient ρ between the scores is given in the plot, and a line of best fit (gray dotted line) is given by a robust linear regression.

 10.1371/journal.pdig.0000324.g006 Traditional evaluation metric (<italic>x</italic>-axis) and weighted accuracy metric (<italic>y</italic>-axis) scores for the final Challenge entries with the murmur detection task on the hidden test set.

AUROC is the area under the receiver operating character curve, AUPRC is the area under the precision-recall curve, F-measure is the macro-averaged F-measure, and accuracy is the traditional accuracy metric. Each point shows an entry, and the shading of each point shows whether the entry was an official entry (dark red points) that satisfied all of the Challenge rules or an unofficial entry (light red points) that did not. The Spearman correlation coefficients ρ between the metrics are given in the individual plots, and a line of best fit (gray dotted line) is given by a robust linear regression.

 10.1371/journal.pdig.0000324.g007 Traditional evaluation metric (<italic>x</italic>-axis) and cost metric (<italic>y</italic>-axis) scores for the final Challenge entries with the clinical outcome identification task on the hidden test set.

AUROC is the area under the receiver operating character curve, AUPRC the is area under the precision-recall curve, F-measure is the traditional F-measure, and accuracy is the traditional accuracy metric. Each point shows an entry, and the shading of each point shows whether the entry was an official entry (dark red points) that satisfied all of the Challenge rules or an unofficial entry (light red points) that did not. The Spearman correlation coefficients ρ between the metrics are given in the individual plots, and a line of best fit (gray dotted line) is given by a robust linear regression.

For the official rankings, we required the teams to have successful unofficial and official phase entries, to have an accepted CinC abstract, to publicly share their CinC proceedings paper preprint by the CinC preprint submission deadline, to have an accepted final CinC proceedings paper by the CinC final paper submission deadline, and to license their code under an open-source or similarly permissive license. We also allowed teams without a successful unofficial phase entry and/or accepted CinC abstract who had a successful, high-scoring official phase entry to submit a CinC abstract after the original abstract submission deadline as a “wild card” team. The wild card abstract and proceedings paper were otherwise subject to the same review process as for the other participants.

We imposed these requirements to support the dissemination of research into the research community. Of the 53 working entries, a total of 40 teams were officially ranked. Tables 5 and 6 summarize the traditional and Challenge evaluation metrics for the officially ranked entries for the murmur detection and clinical outcome identification tasks, respectively, on the hidden test set.

To assess the robustness of the algorithms, we tested whether they actually learned from the training set; algorithms that cannot learn from new data have more limited use. To do so, we trained them on a subset of the training set with permuted labels and ran the retrained models on the validation and test sets with the original labels; we did not change any of the publicly available training data or labels but simply the size of the training set and the relationship between the data and the labels in the training set. Algorithms that truly learned from the training set performed much worse when retrained on the modified training set, but algorithms that did not learn from the training set performed the same or better on the modified training set. Only 35 of the 53 working entries were robust enough for us to complete this process. The remaining 18 working entries either crashed or achieved the same or a better score when retrained on the modified training set, suggesting that they were insensitive to the training set labels. While we encouraged teams to submit robust code, we did not share the details about how we would assess the robustness of their algorithms to encourage them to make their code robust in as many ways as possible. Therefore, since we did not provide advance notice of the exact requirements for robustness, we did not disqualify teams whose algorithms failed this test.

 Voting algorithms

While some entries had better overall performance than others, an entry with a lower overall performance may perform better on certain patient subgroups than another entry with a stronger overall performance. Therefore, an algorithm that can learn the different strengths of different algorithms can improve on the overall performance of the individual entries. To test this concept, we developed multiple voting algorithms to attempt to leverage the diversity of Challenge entries.

For the voting algorithms, we considered the discrete model outputs (see Tables 3 and 4) from the 39 algorithms from teams that were officially ranked for both the murmur detection and clinical outcome tasks (see Tables 5 and 6). We considered these tasks separately, and we used the relevant Challenge scoring metric to evaluate the voting algorithms for each task: the weighted accuracy metric (1) for the murmur detection task and the cost metric (8) for the clinical outcome identification task. We used the common gradient-boosting trees (GBT) and random forests (RF) models for the voting algorithms [15, 16, 17].

We developed and evaluated the GBT and RF voting algorithms using the following procedure. First, we trained the GBT and RF voting algorithms on the k = 1, 2, …, 39 best-performing teams on the training set. Next, we chose k to be the value that achieved the best performance on the validation set; this step identified k = 14 for the GBT models and k = 2 for the RF models on the murmur detection task and k = 14 for the GBT models and k = 4 for the RF models on the clinical outcome identification task. Finally, we evaluated the resulting voting algorithms on the test set.

The GBT and RF voting algorithms performed slightly better than the highest-ranked entry for the murmur detection task (weighted accuracy metric of 0.790 and 0.789, respectively vs. 0.780; see Table 5) and performed slightly worse than the highest-ranked entries for the clinical outcome identification task (cost of 11357 and 11687, respectively vs. 11144; see Table 6). In each case, the voting algorithms had comparable performance to the highest ranked individual algorithms, but they did not significantly outperform them in either task by any of the traditional or novel scoring metrics that we considered.

10.1371/journal.pdig.0000324.t005 Scores of the officially ranked methods on the test set for the murmur detection task.

AUROC is the area under the receiver operating characteristic curve, AUPRC is the area under the precision-recall curve, F-measure is the macro-averaged F-measure, accuracy is the traditional accuracy metric, and weighted accuracy is the weighted accuracy metric (1), i.e., the metric used to rank algorithms for the murmur detection task.

Rank Team AUROC AUPRC F-measure Accuracy Weighted Accuracy
* Voting—GBT 0.677 0.465 0.660 0.845 0.790
** Voting—RF 0.775 0.589 0.650 0.832 0.789
1 HearHeart [19] 0.884 0.716 0.619 0.801 0.780
2 CUED_Acoustics [20] 0.757 0.528 0.623 0.763 0.776
2 HearTech+ [21] 0.771 0.561 0.647 0.822 0.776
4 PathToMyHeart [22] 0.880 0.684 0.686 0.778 0.771
5 CAU_UMN [23] 0.763 0.621 0.521 0.786 0.767
6 Care4MyHeart [24] 0.891 0.717 0.695 0.851 0.757
6 SmartBeatIT [25] 0.873 0.707 0.633 0.809 0.757
8 CeZIS [26] 0.804 0.587 0.586 0.761 0.756
9 ISIBrno-AIMT [27] 0.897 0.746 0.555 0.839 0.755
9 Murmur Mia! [28] 0.867 0.688 0.592 0.771 0.755
11 PhysioDreamfly [29] 0.885 0.723 0.652 0.799 0.752
12 Heart2Beat [30] 0.849 0.650 0.579 0.738 0.751
13 Listen2YourHeart [31] 0.836 0.656 0.597 0.706 0.737
14 Revenger [32] 0.776 0.611 0.559 0.843 0.736
15 matLisboa [33] 0.269 0.248 0.610 0.807 0.735
15 uke-cardio [34] 0.890 0.735 0.597 0.790 0.735
17 One_Heart_Health [35] 0.756 0.597 0.530 0.820 0.729
18 MetaHeart [36] 0.868 0.691 0.548 0.834 0.723
19 Team_IIITH [37] 0.725 0.584 0.536 0.813 0.708
20 prna [38] 0.723 0.579 0.489 0.759 0.694
21 HCCL [39] 0.717 0.569 0.571 0.769 0.690
22 amc-sh [40] 0.814 0.658 0.595 0.803 0.688
23 JUST_IT_Academy1 [41] 0.797 0.610 0.572 0.757 0.671
23 LSMU [42] 0.741 0.584 0.523 0.824 0.671
25 listNto_urHeart [43] 0.819 0.627 0.565 0.780 0.668
26 USST_Med [44] 0.779 0.593 0.524 0.803 0.642
27 PCGPAW [45] 0.890 0.721 0.522 0.824 0.637
28 Melbourne_Kangas [46] 0.789 0.599 0.499 0.725 0.632
29 SeaCrying [47] 0.798 0.612 0.461 0.447 0.601
30 Simulab [48] 0.711 0.530 0.398 0.497 0.593
31 MainLab [49] 0.754 0.537 0.557 0.734 0.583
32 Leicester Fox [50] 0.764 0.608 0.449 0.782 0.536
33 Eagles [51] 0.357 0.276 0.115 0.208 0.525
33 fly_h [52] 0.614 0.481 0.444 0.778 0.525
33 lubdub [53] 0.500 0.333 0.115 0.208 0.525
36 AKSJ_97BSc [54] 0.655 0.425 0.425 0.491 0.494
37 UKJ_FSU [55] 0.576 0.406 0.379 0.730 0.458
38 BrownBAI [56] 0.571 0.367 0.361 0.526 0.406
39 Bear_FH [57] 0.688 0.488 0.328 0.727 0.402
40 MobiHealth [58] 0.562 0.367 0.369 0.606 0.374
10.1371/journal.pdig.0000324.t006 Scores of the officially ranked methods on the test set for the clinical outcome identification task.

AUROC is the area under the receiver operating characteristic curve, AUPRC is the area under the precision-recall curve, F-measure is the traditional F-measure, accuracy is the traditional accuracy metric, and cost is the cost-based evaluation metric (8), i.e., the metric used to rank algorithms for the clinical outcome identification task.

Rank Team AUROC AUPRC F-measure Accuracy Cost
1 CUED_Acoustics [20] 0.693 0.715 0.549 0.602 11144
* Voting-GBT 0.583 0.553 0.546 0.587 11357
2 prna [38] 0.691 0.706 0.536 0.587 11403
** Voting-RF 0.625 0.584 0.596 0.606 11687
3 Melbourne_Kangas [46] 0.663 0.667 0.528 0.568 11735
4 CeZIS [26] 0.614 0.579 0.511 0.560 11916
5 CAU_UMN [23] 0.660 0.653 0.505 0.562 11933
6 HCCL [39] 0.605 0.579 0.532 0.562 11943
7 Listen2YourHeart [31] 0.627 0.607 0.512 0.558 11946
8 uke-cardio [34] 0.665 0.658 0.509 0.556 11990
9 HearTech+ [21] 0.547 0.527 0.508 0.551 12069
10 HearHeart [19] 0.593 0.605 0.528 0.556 12110
11 Heart2Beat [30] 0.636 0.625 0.560 0.572 12244
12 ISIBrno-AIMT [27] 0.586 0.567 0.505 0.541 12313
13 UKJ_FSU [55] 0.565 0.582 0.481 0.539 12373
14 Simulab [48] 0.375 0.421 0.503 0.537 12419
15 MetaHeart [36] 0.500 0.500 0.511 0.537 12536
16 matLisboa [33] 0.644 0.654 0.545 0.558 12593
17 PathToMyHeart [22] 0.566 0.567 0.460 0.535 12637
18 PhysioDreamfly [29] 0.602 0.626 0.486 0.518 12831
19 Revenger [32] 0.594 0.604 0.442 0.520 12944
20 amc-sh [40] 0.593 0.594 0.512 0.528 13002
21 MainLab [49] 0.572 0.572 0.517 0.528 13259
22 Team_IIITH [37] 0.616 0.610 0.533 0.541 13264
23 One_Heart_Health [35] 0.587 0.594 0.531 0.539 13283
24 JUST_IT_Academy1 [41] 0.624 0.631 0.559 0.562 13394
25 SmartBeatIT [25] 0.647 0.655 0.593 0.593 13815
26 Leicester Fox [50] 0.662 0.646 0.620 0.621 13844
27 listNto_urHeart [43] 0.489 0.490 0.495 0.505 13866
28 Murmur Mia! [28] 0.682 0.688 0.382 0.528 14228
29 Care4MyHeart [24] 0.624 0.632 0.597 0.597 14410
30 USST_Med [44] 0.623 0.621 0.583 0.583 14529
31 lubdub [53] 0.560 0.551 0.345 0.509 14905
32 PCGPAW [45] 0.500 0.500 0.337 0.507 15083
32 SeaCrying [47] 0.500 0.500 0.337 0.507 15083
34 LSMU [42] 0.325 0.399 0.344 0.388 15402
35 Bear_FH [57] 0.578 0.590 0.549 0.551 15982
35 fly_h [52] 0.578 0.590 0.549 0.551 15982
37 AKSJ_97BSc [54] 0.549 0.550 0.518 0.520 16427
38 BrownBAI [56] 0.487 0.486 0.497 0.499 16773
39 MobiHealth [58] 0.523 0.512 0.503 0.520 18754
Discussion

Cardiac auscultation is one of the most cost-effective tools for helping clinicians to identify heart murmurs, and the CirCor Digiscope dataset enriches our understanding cardiac auscultation within a resource-constrained environment. However, despite the novelty and value of this dataset, it, like every dataset, has limitations.

No ages were available for the pregnant individuals in the CirCor DigiScope dataset. It was unclear to the teams if the pregnant individuals belonged to the pediatric age group of the rest of the patients, or if they had a different set of exclusion criteria from the other patients, potentially limiting the performance and appropriateness of models that use this dataset on pregnant individuals.

There was a single human annotator for labeling the heart murmurs; we did not have information about the number of human annotators for labeling the clinical outcomes of abnormal or normal heart function. We expect disagreements between annotators, and several annotators are often needed to produce a single, consistent annotation of a health record. Even several annotators may produce disparate annotations [59]. It is likely that different or more annotators would result in different annotations.

Some issues, such as background noises in the recordings and the relatively high heart rates of children, are inherent to the data or to the collection of the data. To preserve the realism of the problem and the diversity of the approaches, we did not remove these issues, enumerate them as potential difficulties, or suggest how to address them; many of the solutions employ different strategies to address different problems with the data.

Some heart murmurs are pathological, indicating a physiological problem that requires monitoring and/or intervention. Other heart murmurs are innocent, with sounds resulting from normal blood flow, and do not require treatment. However, the human annotator did not identify which heart murmurs were pathological or innocent; the clinical outcomes, which reflect a full examination of the patient, are the strongest evidence in the dataset for characterizing the severity of any heart murmurs or underlying cardiac disease. Therefore, we only evaluated the ability of algorithms to identify heart murmurs, not their ability to identify pathological heart murmurs or differentiate between pathological and innocent heart murmurs, but this task is still valuable for screening for cardiovascular disease.

The definition of a task and the choice of evaluation metrics for quantifying an algorithm’s performance on the task affect the actual and perceived clinical relevance of the algorithms [14]. The definitions of the Challenge evaluation metrics (1) and (8) are attempts to make the evaluation metrics, and therefore the algorithms developed for these metrics, more clinically relevant. The correlation in performance between the traditional and Challenge metrics demonstrates that methods that perform better by one metric tended to perform better by another metric (at least for the murmur detection task), but the best-scoring method by one metric was often different from the best best-scoring method by another metric, motivating the careful consideration of metrics; see Figs 6 and 7. We also recognize that these metrics are imperfect descriptions of clinical needs; in particular, healthcare cost can be a poor proxy for health needs [13, 14].

The relatively poor performance of the methods by all metrics on the clinical outcome identification task suggests the difficulty of performing this task using the PCG recordings alone; we note that echocardiography is a standard diagnostic tool for assessing cardiac function, which is a more expensive and less accessible modality than phonocardiography.

The voting algorithms had only modest performance improvements, if any, over the individual algorithms. Indeed, the voting algorithms did not use any data-derived features, which could help to provide context by associating different algorithms with different patient subgroups. The features for the voting algorithms included the top k algorithms, even when including a lower-ranked entry with a worse overall score or excluding a higher-ranked entry with a better overall score would improve the performance of the voting algorithm. This experiment only considered GBT and RF models, but other types of models could potentially achieve better performance. The training procedure for the GBT and RF models did not use hyperparameter optimization (beyond the selection of the number k of individual algorithms), limiting their performance as well. Also, unlike the Challenge teams, we had access to the test set, but, like the Challenge teams, our formal training procedure did not use it. Despite these limitations, the ability of the voting algorithms to slightly improve on the performance on the top-ranked algorithms for the murmur detection task, but not the clinical outcome identification task, also suggests the differences in difficulty between the two related tasks considered by the Challenge.

 Conclusions

The George B. Moody PhysioNet Challenge explored the potential for algorithmic pre-screening for heart murmurs and abnormal cardiac function in resource-constrained environments. We invited the Challenge participants to design working, open-source algorithms for identifying heart murmurs and clinical outcomes from PCG recordings, resulting in 53 working implementations of different algorithmic approaches for interpreting PCGs. A voting approach to combining the diverse approaches resulted in a superior performance over the individual algorithms for murmur detection but not for clinical outcome identification.

By reducing human screening of patients with normal cardiac function, algorithms can lower healthcare costs and increase the effective screening capacity for patients with abnormal cardiac function. The cost function proposed in this Challenge could be the basis for cost-effective screening that balances both the financial and health burden of correctly or incorrectly diagnosing patients. However, it will be important to optimize the proposed cost function for a given healthcare system or population, since disease prevalence, financial resources, and healthcare resources can differ enormously in different settings.

 Supporting information Additional scoring metrics.

(PDF)

 Mathematical derivation of the cost-based scoring metric.

(PDF)

None of the funding entities influenced the design of the Challenge or provided data for the Challenge. The content of this manuscript is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.

 References Amin DS, Fethi BR. Features for heartbeat sound signal normal and pathological. Recent Patents on Computer Science. 2008;1(1):1–8. doi: 10.2174/1874479610801010001 Singh J, Anand RS. Computer aided analysis of phonocardiogram. Journal of Medical Engineering & Technology. 2007;31(5):319–323. doi: 10.1080/03091900500282772 17701776 Marcus G, Vessey J, Jordan MV, Huddleston M, McKeown B, Gerber IL, et al. Relationship Between Accurate Auscultation of a Clinically Useful Third Heart Sound and Level of Experience. Archives of Internal Medicine. 2006;166(6):617–622. doi: 10.1001/archinte.166.6.617 16567599 Ishmail AA, Wing S, Ferguson J, Hutchinson TA, Magder S, Flegel KM. Interobserver Agreement by Auscultation in the Presence of a Third Heart Sound in Patients with Congestive Heart Failure. Chest. 1987;91(6):870–873. doi: 10.1378/chest.91.6.870 Vermarien H. Phonocardiography. In: Webster JG, editor. Encyclopedia of Medical Devices and Instrumentation. vol. 5. 2nd ed. John Wiley & Sons, Ltd; 2006. p. 278–290. Oliveira JH, Renna F, Costa P, Nogueira D, Oliveira C, Ferreira C, et al. The CirCor DigiScope Dataset: From Murmur Detection to Murmur Classification. IEEE Journal of Biomedical and Health Informatics. 2021;26(6):2524–2535. doi: 10.1109/JBHI.2021.3137048 Liu C, Springer D, Li Q, Moody B, Juan RA, Chorro FJ, et al. An open access database for the evaluation of heart sound algorithms. Physiological Measurement. 2016;37(12):2181–2213. doi: 10.1088/0967-3334/37/12/2181 27869105 Oliveira J, Renna F, Mantadelis T, Coimbra M. Adaptive Sojourn Time HSMM for Heart Sound Segmentation. IEEE Journal of Biomedical and Health Informatics. 2019;23(2):642–649. doi: 10.1109/JBHI.2018.2841197 29993729 Renna F, Oliveira J, Coimbra MT. Deep Convolutional Neural Networks for Heart Sound Segmentation. IEEE Journal of Biomedical and Health Informatics. 2019;23(6):2435–2445. doi: 10.1109/JBHI.2019.2894222 30668487 Freeman AR. The Clinical Significance of the Systolic Murmur. Annals of Internal Medicine. 1933;6(11):1371. doi: 10.7326/0003-4819-6-11-1371 Goldberger AL, Amaral LA, Glass L, Hausdorff JM, Ivanov PC, Mark RG, et al. PhysioBank, PhysioToolkit, and PhysioNet: Components of a New Research Resource for Complex Physiologic Signals. Circulation. 2000;101(23):e215–e220. doi: 10.1161/01.CIR.101.23.e215 10851218 Reyna MA, Josef C, Jeter R, Shashikumar SP, Westover MB, Nemati S, et al. Early Prediction of Sepsis from Clinical Data: the PhysioNet/Computing in Cardiology Challenge 2019. Critical Care Medicine. 2019;48:210–217. doi: 10.1097/CCM.0000000000004145 Mullainathan S, Obermeyer Z. On the Inequity of Predicting A While Hoping for B. In: AEA Papers and Proceedings. vol. 111; 2021. p. 37–42. doi: 10.1257/pandp.20211078 Reyna MA, Nsoesie EO, Clifford GD. Rethinking Algorithm Performance Metrics for Artificial Intelligence in Diagnostic Medicine. JAMA. 2022;328(4):329–330. doi: 10.1001/jama.2022.10561 35802382 Friedman JH. Greedy function approximation: a gradient boosting machine. Annals of Statistics. 2001; p. 1189–1232. Chen T, Guestrin C. XGBoost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2016. p. 785–794. Breiman L. Random forests. Machine Learning. 2001;45:5–32. Reyna M, Kiarashi Y, Andoni E, Oliveira J, Renna F, Gu A, et al. Heart Murmur Detection from Phonocardiogram Recordings: The George B. Moody PhysioNet Challenge 2022. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Lu H, Yip J, Steigleder T, Grießhammer S, Jami NVSJ, Eskofier B, et al. A Lightweight Robust Approach for Automatic Heart Murmurs and Clinical Outcomes Classification from Phonocardiogram Recordings. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. McDonald A, Gales M, Agarwal A. Detection of Heart Murmurs in Phonocardiograms with Parallel Hidden Semi-Markov Models. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Xu Y, Bao X, Lam HK, Kamavuako E. Hierarchical Multi-Scale Convolutional Network for Murmurs Detection on PCG Signals. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Krones F, Walker B, Mahdi A, Kiskin I, Lyons T, Parsons G. Dual Bayesian ResNet: A Deep Learning Approach to Heart Murmur Detection. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Lee J, Kang T, Kim N, Han S, Won H, Gong W, et al. Deep Learning Based Heart Murmur Detection using Frequency-time Domain Features of Heartbeat Sounds. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Alkhodari M, Azman S, Hadjileontiadis L, Khandoker A. Ensemble Transformer-Based Neural Networks Detect Heart Murmur In Phonocardiogram Recordings. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Monteiro S, Fred A, Silva H. Detection of Heart Sound Murmurs and Clinical Outcome with Bidirectional Long Short-Term Memory Networks. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Ľubomír Antoni, Bruoth E, Szabari A, Vozáriková G, Bugata P, Jr PB, et al. Murmur Identification Using Supervised Constrastive Learning. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Nejedly P, Pavlus J, Smisek R, Koscova Z, Vargova E, Viscor I, et al. Classification of heart murmurs using an ensemble of residual CNNs. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Summerton S, Wood D, Murphy D, Redfern O, Benatan M, Kaisti M, et al. Two-stage Detection of Murmurs from Phonocardiograms using Deep and One-class Methods. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Bai Z, Yan B, Chen X, Wu Y, Wang P. Murmur Detection and Clinical Outcome Classification Using a VGG-like Network and Combined Time-Frequency Representations of PCG Signals. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Rohr M, Müller B, Dill S, Güney G, Antink CH. Two-Stage Multitask-Learner for PCG Murmur Location Detection. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Ballas A, Papapanagiotou V, Delopoulis A, Diou C. Listen to your heart: A self-supervised approach for detecting murmur in heart-beat sounds. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. WEN H, Kang J. Searching for Effective Neural Network Architectures for Heart Murmur Detection from Phonocardiogram. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Costa J, Rodrigues R, Couto P. Multitask and Transfer Learning for Murmur Detection in Heart Sounds. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Knorr M, Bremer J, Schnabel R. Using Mel-Spectrograms and 2D-CNNs to detect Murmurs in Variable Length Phonocardiograms. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Araujo M, Zeng D, Palotti J, Hu X, Shi Y, Pyles L, et al. Maiby’s Algorithm: A Two-stage Deep Learning approach for Murmur Detection in Mel Spectrograms for Automatic Auscultation of Congenital Heart Disease. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Xia P, Yao Y, Liu C, Zhang H, Wang Y, Xu L, et al. Heart Murmur Detection from Phonocardiogram Based on Residual Neural Network with Classes Distinguished Focal Loss. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Venkataramani VV, Garg A, Priyakumar UD. Modified Variable Kernel Length ResNets for Heart Murmur Detection and Clinical Outcome Prediction using Multi-positional Phonocardiogram Recording. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Chang Y, Liu L, Antonescu C. Multi-Task Prediction of Murmur and Outcome from Heart Sound Recordings. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Kim J, Park G, Suh B. Classification of Phonocardiogram Recordings using Vision Transformer Architecture. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Shin JM, Kim HS, Seo WY, Kim SH. Learning Time-Frequency Representations of Phonocardiogram for Murmur Detection. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Peng G, Zou H, Wang J. Classification of phonocardiograms using residual convolutional neural network and MLP. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Petrolis R, Paukstaitiene R, Rudokaite G, Macas A, Grigaliunas A, Krisciukaitis A. Convolutional neural network aproach for heart MurMur detection in auscultation signals using wavelet transform based features. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Gemke P, Spicher N, Kacprowski T. An LSTM-based Listener for Early Detection of Heart Disease. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Hu L, Cai W, Li X, Li J. Detection of murmurs from heart sound recordings with deep residual networks. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Warrick P, Afilalo J. Phonocardiographic Murmur Detection by Scattering-Recurrent Networks. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Imran Z, Grooby E, Sitaula C, Malgi V, Aryal S, Marzbanrad F. A Fusion of Handcrafted Features and Deep Learning Classifiers for Heart Murmur Detection. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Wang X, Fan F, Gao H, Zhang S, Yang C, Li J, et al. Beat-wise uncertainty learning for murmur detection in heart sounds. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Singstad BJ, Bongo L, Johnsen M, Ravn J, Gitau A, Schirmer H. Phonocardiogram Classification Using 1-Dimensional Inception Time Convolutional Neural Networks. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Choi S, Seo HC, Choi K, Yoon G, Joo S. Murmur Classification with U-net State Prediction. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Li X, Schlindwein F, Ng GA. Transfer Learning in Heart Sound Classification using Mel spectrogram. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Cornely A, Mirsky G. Heart Murmur Detection Using Wavelet Time Scattering and Support Vector Machines. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Gao Y, Qiao L, Li Z. Heart Murmur Detection of PCG Using ResNet with Selective Kernel Convolution. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Ding J, Li JJ, Xu M. Classification of Murmurs in PCG Using Combined Frequency Domain and Physician Inspired Features. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Jalali K, Saket MA, Noorzadeh S. Heart Murmur Detection and Clinical Outcome Prediction using Multilayer Perceptron Classifier. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Stein G, Festag S, Büchner T, Shadaydeh M, Denzler J, Spreckelsen C. Outcome Prediction and Murmur Detection in Sets of Phonocardiograms by a Deep Learning-Based Ensemble Approach. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Rudman W, Merullo J, Mercurio L, Eickhoff C. ACQuA: Anomaly Classification with Quasi-Attractors. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Fan P, Shu Y, Han Y. Transformer embedded with learnable filters for heart murmur detection. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Bondareva E, Han J, Xia T, Mascolo C. Towards Uncertainty-Aware Murmur Detection in Heart Sounds via Tandem Learning. In: 2022 Computing in Cardiology (CinC). vol. 49. IEEE; 2022. p. 1–4. Clifford GD, Liu C, Moody B, Lehman LwH, Silva I, Li Q, et al. AF Classification from a Short Single Lead ECG Recording: the PhysioNet/Computing in Cardiology Challenge 2017. In: 2017 Computing in Cardiology (CinC). vol. 44. IEEE; 2017. p. 1–4. 10.1371/journal.pdig.0000324.r001 Decision Letter 0 Celi Leo Anthony Editor-In-Chief 2023 Leo Anthony Celi This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 0

19 May 2023

PDIG-D-23-00131

Heart murmur detection from phonocardiogram recordings: The George B. Moody PhysioNet Challenge 2022

PLOS Digital Health

Dear Dr. Reyna,

Thank you for submitting your manuscript to PLOS Digital Health. After careful consideration, we feel that it has merit but does not fully meet PLOS Digital Health's publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript within 30 days Jul 18 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at digitalhealth@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pdig/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

* A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

* A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

* An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Leo Anthony Celi, MD MS MPH

Editor-In-Chief

PLOS Digital Health

Journal Requirements:

1. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article's retracted status in the References list and also include a citation and full reference for the retraction notice.

2. Please send a completed 'Competing Interests' statement, including any COIs declared by your co-authors. If you have no competing interests to declare, please state "The authors have declared that no competing interests exist". Otherwise please declare all competing interests beginning with twhe statement "I have read the journal's policy and the authors of this manuscript have the following competing interests:"

Additional Editor Comments (if provided):

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Does this manuscript meet PLOS Digital Health’s publication criteria? Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe methodologically and ethically rigorous research with conclusions that are appropriately drawn based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

--------------------

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

--------------------

3. Have the authors made all data underlying the findings in their manuscript fully available (please refer to the Data Availability Statement at the start of the manuscript PDF file)?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

--------------------

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS Digital Health does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

--------------------

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The manuscript presents results from the “Heart murmur detection from phonocardiogram recordings: The George B. Moody PhysioNet Challenge 2022”. It is well written and easy to follow. The manuscript uses up to date references; which are all well linked within the document. All reference materials have been provided.

However, the following could be addressed.

- The abstract is great, but consider adding a paragraph about the results (submitted algorithms) from the challenge. The abstract presents the background information; and how the challenge was conducted, but is very silent about the results (which algorithm performed best from the challenge). You might also consider mentioning the aim of this opinion paper in the abstract, to provide a reader with the general picture of the paper.

- The usage of the phrase “This year’s challenge” appears misleading in some paragraphs. May be consider using “The 2022 year’s challenge”; so that it is easy for the reader not to confuse it with the current year.

- The paragraph “while algorithms that did not learn from the training set performed the same or better” on page 12/24 is incomplete.

- The paragraph “requires monitoring and/or intervention” on page 14/24 is incomplete.

- Check paragraph “models and k = 2 for the RF models.” on page 13/24

Reviewer #2: Thank you for the opportunity to review this interesting manuscript. This purpose of the paper was to report findings from The 2022 George B. Moody PhysioNet Challenge (formerly the PhysioNet/Computing in Cardiology Challenge). This challenge invited teams to develop fully automated approaches for detecting heart murmurs and abnormal cardiac function from PCG recordings. The investigators sourced and shared PCG recordings for up to four auscultation locations from a largely pediatric population in Brazil, and teams were asked to identify both heart murmurs and the clinical outcomes of a full diagnostic screening from the recordings. The Challenge explored the diagnostic potential of algorithmic approaches for interpreting PCG recordings. Below are suggestions designed to enhance the readability, clinical understanding and reproducibility of this already well written manuscript.

The format of the paper made it much easier to review this manuscript – thank you.

Title – consider adding “Pediatric” to the title.

Perhaps this was missed by the reviewer, but I did not see key words, are they required?

A limitation of this paper is that the sample was not clearly described. The authors state that the PCG recording were from pediatric patients, yet pregnant individuals were mentioned as having contributed recordings. One might assume if pregnant women were included the PCG recording were obtained from a fetus on the abdomen of the mother? Among pediatric patients one might assume the PCG recordings were obtain from the child’s chest? How might this impact recordings (chest vs abdomen) is this is how the PCG was recorded?

While it may be outside of the scope of this paper one would also imagine that PCG recordings are impacted by heart rate, which in pediatric patients (in your study you included Neonate (birth to 27 days), Infant (28 days to 1 year), Child (1 to 11 years), Adolescent (12 to 18 years), or fetus, in the case of pregnant individuals, which would be very challenging since heart rate is so fast in younger children. Was heart rate considered or accounted for? Again, this may not be appropriate to discuss in this paper, but could be mentioned.

In the Introduction, second sentence the term “technology is used. Yet a stethoscope is a device, which may be a more appropriate term.

Introduction second paragraph – the statements regarding “within the valves” is not entirely accurate. Suggest using “around,” rather than “within” as blood flow is disrupted around an incompetent valve. The authors may want to delete the statement “…..and by the turbulence of blood flow within the valves” in the first sentence and end it after “….cardiac cycle.” Then make the suggested adjustment in the next sentence with refers to the pathology.

Figure 1 – the AV and PV locations are both at the location of the second intercostal space yet the image shows them slightly off, suggesting that the PV is in the third intercostal space.

Page 4 second paragraph – Consider adding (described below) to the sentence “Each patient’s PCGs and clinical notes were also annotated for murmurs.” This addition would help the reader understand that this description will be discussed further down in the paper.

Can “resource constrained” be defined i.e., rural vs urban, socioeconomic etc. this was not entirely clear.

Page 6 sentence that reads “………………important issue in the Section .” seem to be missing the section #.

For the text description for Figures 3 and 4 consider bolding the text for “per-patient” and “per-screening” to add clarity to the description.

--------------------

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

Do you want your identity to be public for this peer review? If you choose “no”, your identity will remain anonymous but your review may still be made public.

For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Dr. Wasswa William | Mbarara University of Science and Technology

Reviewer #2: No

--------------------

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: PDIG-D-23-00131_comments.pdf

 10.1371/journal.pdig.0000324.r002 Author response to Decision Letter 0 Submission Version 1

23 Jun 2023

Submitted filename: 2022 Challenge Manuscript Review Response.pdf

 10.1371/journal.pdig.0000324.r003 Decision Letter 1 Celi Leo Anthony Editor-In-Chief 2023 Leo Anthony Celi This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 1

29 Jun 2023

PDIG-D-23-00131R1

Heart murmur detection from phonocardiogram recordings: The George B. Moody PhysioNet Challenge 2022

PLOS Digital Health

Dear Dr. Reyna,

Thank you for submitting your manuscript to PLOS Digital Health. After careful consideration, we feel that it has merit but does not fully meet PLOS Digital Health's publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript within 30 days Jul 29 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at digitalhealth@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pdig/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

* A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

* A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

* An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Leo Anthony Celi, MD MS MPH

Editor-In-Chief

PLOS Digital Health

Journal Requirements:

1. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article's retracted status in the References list and also include a citation and full reference for the retraction notice.

2. Please send a completed 'Competing Interests' statement, including any COIs declared by your co-authors. If you have no competing interests to declare, please state "The authors have declared that no competing interests exist". Otherwise please declare all competing interests beginning with twhe statement "I have read the journal's policy and the authors of this manuscript have the following competing interests:"

Additional Editor Comments (if provided):

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

 10.1371/journal.pdig.0000324.r004 Author response to Decision Letter 1 Submission Version 2

30 Jun 2023

Submitted filename: 2022 Challenge Manuscript Review Response.pdf

 10.1371/journal.pdig.0000324.r005 Decision Letter 2 Celi Leo Anthony Editor-In-Chief 2023 Leo Anthony Celi This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 2

10 Jul 2023

Heart murmur detection from phonocardiogram recordings: The George B. Moody PhysioNet Challenge 2022

PDIG-D-23-00131R2

Dear Prof. Reyna,

We are pleased to inform you that your manuscript 'Heart murmur detection from phonocardiogram recordings: The George B. Moody PhysioNet Challenge 2022' has been provisionally accepted for publication in PLOS Digital Health.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow-up email from a member of our team. 

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they'll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact digitalhealth@plos.org.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Digital Health.

Best regards,

Leo Anthony Celi, MD MS MPH

Editor-In-Chief

PLOS Digital Health

***********************************************************

Reviewer Comments (if any, and for reference):