This is a retrospective, observational, multicenter cohort study. Our primary objective was to develop and internally validate a prognostic mortality risk model.

Hackensack Meridian Health network (HMH) had established an observational database drawn from the electronic health records (EHR) of hospitalized COVID-19 patients as described below (Clinicaltrials.gov Identifier: NCT04347993). Patients were included in the database based on the following inclusion and exclusion criteria: 1) Positive SARS-CoV-2 diagnosis by reverse-transcriptase polymerase chain reaction and 2) Hospitalization at one of HMH’s 13-hospitals within the time frame of March 1, 2020 until April 22, 2020. For the purposes of generating the mortality-risk model we also excluded 1) pregnancy, 2) enrollment in a randomized clinical trial, and 3) died on the day of admission to the hospital. We accessed the data on June 10, 2020.

Institutional Review Board (IRB) approval was obtained for access to the prospective observational database. Informed consent was waived by the IRB as this project represented a non-interventional study utilizing de-identified data within the database.

We collected data from HMH’s EHR (Epic, Verona WI) which is utilized throughout the network. Hospitalized patients throughout HMH were flagged by the EHR if SARS-CoV-2 testing was positive. These EHR-generated reports served as our eligible cohort to sample. Demographic, clinical characteristics, treatments, and outcomes were manually abstracted by research nurses and physicians from the John Theurer Cancer Center at Hackensack University Medical Center. Assigning patients to our data team occurred in real-time, and not randomized. Data abstracted by the team were entered using REDCap (Research Electronic Data Capture, Vanderbilt University). Quality control was performed by physicians (AI, SLG) overseeing nurse or physician abstraction.

Demographic information was collected by an electronic facesheet, with gender, race, or ethnicity self-reported. Academic centers were defined as quaternary referral centers with accredited residency, fellowship, and medical student programs. Nursing home or rehabilitation patients, if diagnosed prior to hospitalization, were defined as ambulatory patients. Comorbidities were defined as diagnosed prior to hospitalization for COVID-19. History of hypertension, diabetes, chronic lung disease (COPD or asthma), cancer, coronary artery disease, cerebrovascular disease, renal failure, and rheumatologic disease were abstracted from provider notes or medical history sections found within the EHR. If not listed in the patient’s record, the comorbidity was recorded as absent.

Presenting clinical data was abstracted from thorough review of unstructured notes as well as structured data. Hospital readmissions were included in the original admission, with baseline data used from the initial hospitalization. If multiple positive or indeterminate results were found in a patient’s record for SARS-CoV-2, the first positive test defined the date of diagnosis.

Twenty-two factors were considered as potentially predictive of mortality (Table 1). Of these 21 are categorical and 19 of these are dichotomous. Age is the only continuous factor. Multiple prior analyses have suggested that age is a strong predictor of COVID-19 related mortality [1–3]. The function of age that best represents its impact on mortality however is not clear. Although a step function categorized within intervals of age is a potential variable, risk can increase greatly even over as little as 5 years. Therefore, before incorporating age into the statistical procedure described below we tabulated mortality rate into 20 5-year intervals ranging from 0–4 to 100–104 and chose the power function of age that fit the data best in the sense of least squares, choosing among powers 1.0 (linear), 1.2, 1.4, etc.

The primary outcome measurement was death due to any cause within 40 days of hospital admission. Patients alive at day 40 in the hospital as of May 29, 2020 or discharged alive from the hospital were censored on day 40.

We randomly selected two equally sized halves of the dataset. We built the RS on the first half (the training set) and validated it on the second half (the confirmation set). Restricting to the training set we evaluated whether each of the 22 factors listed in Table 1 contributed to mortality risk in the context of the other factors. To reduce 22-dimensional risk information into a univariate risk score we proceeded in two steps. Step 1 considered each of the factors in a univariate proportional hazards model. Factors that were not statistically significant (p-value > 0.05) were dropped from further consideration for the RS. Step 2 considered the remaining factors in a multivariate proportional hazards model. Factors that were not statistically significant in this model were dropped one at a time, with the least significant factor (largest p-value) dropped from further consideration. The analysis was redone with that factor eliminated from the model. This iteration stopped when all the factors in the model were statistically significant. The RS is the final multivariate model standardized so that its range is from 0 to 10 for all possible patient risk profiles (with maximum age 104). The final step is to calculate the risk ratio (and its 95%) confidence interval) per unit of the RS in a univariate proportional hazards regression.

The model was subsequently validated on the second half of the patient cohort (confirmation set). The primary conclusion of the model building exercise is the statistical significance level of the RS built from the Training Set when applied to the Confirmation Set. As an additional and more detailed confirmation we compared actual mortality within each of 20 equal-sized bins of RS values.

We converted the RS into a patient-level predictive probability of 40-day mortality based on the combined dataset. A product of a proportional hazards regression is “baseline mortality at the mean.” Its value at day 40, defined as “BMM40”, is the probability of death by day 40 for an individual patient whose RS equals the mean. BMM40 is calculated for patients with arbitrary RS under the proportional hazards assumption. Namely, the probability of mortality by day 40 is PM40=1−(1−BMM40)RRRS−MeanRS where RR is the risk ratio for one unit of increase in RS.

However, proportional hazards is a strong assumption. If the assumption does not provide a reasonable fit to the actual mortality rates over the full range of RS then for the patient-level mortality risk model we will instead use an empirically derived estimate of PM40 by fitting a polynomial to the actual mortality within the 20 equally sized bins of RS described above, using the combined datasets (training and confirmatory).

Statistical calculations were carried out using JMP^®, Version 15 (390308). SAS Institute Inc., Cary, NC, 1989–2019.

Data on 3478 hospitalized patients were abstracted for this study. 3308 subjects met inclusion criteria, excluding 43 patients who were pregnant, 88 on clinical trials, and 39 with a death within 1 day of admission. 185 had insufficient data available regarding discharge status for analysis, leaving a final cohort of 3123. Table 2 shows the marginal distributions of outcomes for baseline characteristics and potential risk factors. The median age of the entire cohort was 63 years (interquartile range 51 to 74) with a male predominance (60%). African-Americans comprised 11% of the study. Some comorbidities were common with 53% having hypertension and 31% diabetes. Most comorbidities were rare but 42% of all patients had 3 or more chronic conditions or risk factors. 127 patients (4%) were admitted to the ICU within the first day of hospitalization. Oxygen saturation below 94% was identified in 41%. When measured and recorded in the electronic health record, inflammatory markers were elevated with serum ferritin >1500 ng/mL in 26% and d-dimer >1 mcg/mL in 78% of patients. Therapies varied at the multiple hospitals but included hydroxychloroquine in the majority of patients, tocilizumab in over 200 critically ill patients, high dose corticosteroids in most ICU patients, remdisivir in selected patients on trial, and prone ventilation.

In the training cohort (n = 1561), as of May 29, 2020 there were 336 (22%) deaths, 1066 (68%) successful hospital discharges, and 159 (10%) remained at risk while hospitalized and were censored. In the confirmation cohort (n = 1562), there were 377 (24%) deaths, 1053 (67%) successful hospital discharges, and 132 (8%) remained at risk while hospitalized.

Advanced age was the strongest predictor of 40-day mortality. However, the relationship was nonlinear. After tabulating mortality rates into 20 separate 5-year intervals, the best fit using a least squares model was the cube of age (i.e., age³) and this functional form was used in the risk model development below (Fig 1).

As shown in Table 1, 22 factors were considered in developing the mortality risk score model, with 17 factors passing step one by demonstrating in a univariate proportional hazard model a significance level less than 0.05. These factors were entered into the multivariate proportional hazards model, and after serially eliminating the least significant factors in the order shown in Table 1, 6 factors remained in the mortality risk model: age³, respiratory rate ≥25/minute upon hospital presentation, oxygenation <94% on hospital presentation, and pre-hospital comorbidities of hypertension, coronary artery disease, or chronic renal disease.

The risk score for mortality was constructed utilizing the coefficients from the final multivariate proportional hazards model: RS=4.8939e−6*age3+1.3711*resp+1.1623*oxy+0.4736*hyp+0.6362*cad+0.8520*renal; where age is the age in years as stated on the day of hospitalization, resp equals 1 if the respiratory rate is ≥25/minute on admission (and 0 if less), oxy equals 1 if the oxygenation level is <94% on room air upon admission (and 0 if greater), hyp equals 1 if the patient has a pre-existing hypertension comorbidity (and 0 if not), cad equals 1 if the patient has a pre-existing coronary artery disease comorbidity (and 0 if not), and renal equals 1 if the patient has a pre-existing chronic renal insufficiency comorbidity (and 0 if not). The coefficients were standardized so that the smallest possible Risk Score is 0 and the largest possible is 10. As shown in Table 3, a one unit increase in the risk score resulted in an increased risk ratio for mortality by day 40 after hospitalization of 1.80 (95% CI, 1.72, 1.87) in the combined dataset. Few patients had scores above 7, leading to an unstable model above this cutoff.

As shown in Table 3 the risk ratio for 40-day mortality in the training and confirmatory cohorts are similar, but with a slight and nonsignificant diminution of effect in the confirmation cohort, as expected. Each cohort was subsequently divided into 20 bins (156 patients each, except 3 bins of 157 patients) based on ascending means of risk scores. As shown in Fig 2, the percent mortality by bin increased in similar fashions in the training and confirmatory cohorts.

To translate the risk score to a patient-specific probability of mortality within 40 days of hospitalization for COVID-19 we utilized a proportional hazards model as described above. However, this method tended to overestimate risk for low-risk patients and underestimate risk for higher risk patients (Fig 3).

Therefore, we used an empirically derived estimate of day-40 mortality by fitting a polynomial to actual mortality within the 20 equally sized bins of RS described above, using the combined datasets (training and confirmatory). This yielded a better fit for the model, as shown in Fig 4. Thus the patient-specific day-40 risk of mortality was defined as: −14.258+13.460*RS+1.585*(RS−2.524)2−0.403*(RS−2.524)3.

An online calculator based on risk score has been developed to allow clinicians to enter the values of the 6 variables required for the risk score with automatic calculation of the projected 40-day COVID-19 mortality. (Available at www.HackensackMeridianHealth.org/CovidRS).