In the emergency department (ED), decisions have to be made quickly. Misjudgments and misdiagnoses have adverse consequences, especially if resulting in disposition to lower levels of care or to discharge. Therefore, identification of the severely ill is of utmost importance. The use of triage tools is ubiquitous, but these may lack power in certain patient groups, such as older populations [1] and patients with nonspecific complaints such as weakness or fatigue [2]. Weakness and fatigue are associated with higher in-hospital mortality [3], higher resource use, longer ED length of stay, and higher long-term mortality in older patients [4, 5]. However, the characteristics of ED patients with weakness and/or fatigue are not well understood.

ED physicians’ disease severity rating (DSR)—their immediate and subjective judgment of how ill patients look, typically recorded at triage before any other assessment, particularly before receiving test results—is a promising predictor of adverse outcomes. Several studies found that nurses’ and physicians’ DSR can predict mortality in emergency department patients [6]. Another study found that DSR from phlebotomists can outperform the Danish Emergency Process Triage (DEPT) in predicting mortality [7]. Further research has shown that morbidity can be predicted with computerized algorithms based on both clinical markers and physicians’ DSR even in ED patients with nonspecific complaints [8]. When predicting patient outcomes for patients with nonspecific complaints using DSR, physicians’ accuracy was better than chance [9], and morbidity in these patients could be predicted based on DSR [10].

Our goals were (1) to characterize ED patients with weakness and/or fatigue; to explore (2) to what extent physicians’ disease severity rating (DSR) at triage can predict five key outcomes in patients presenting to the ED with or without weakness and/or fatigue (W|F); (3) how well DSR performs relative to two commonly used benchmark methods, the Emergency Severity Index (ESI) and the Charlson Comorbidity Index (CCI); (4) to what extent DSR provides predictive information beyond ESI, CCI, or their linear combination, i.e., whether ESI and CCI should be used alone or in combination with DSR; and (5) to what extent ESI, CCI, or their linear combination provide predictive information beyond DSR alone, i.e., whether DSR should be used alone or in combination with ESI and / or CCI. For the fourth and fifth goal, we use logistic regression models to explore the predictive power of two or three scores when combined into a new score. Importantly, even though logistic regression models give probability values as their output, we interpret those values only as a score and not as a proper probability because it is not our goal to already propose prognostic prediction models (e.g., estimated coefficients of a logistic regression model), which then—following best-practices [11]—would need to be thoroughly evaluated with regard to further properties, such as Brier score, calibration curves etc. For all these reasons, we restrict ourselves to reporting ROC curve analyses (see Methods below) and will thus not report Brier scores, calibration curves etc. Nevertheless, to ensure reporting transparency, we adhere to the TRIPOD statement for reporting [11] with respect to the results we do present.

We carried out a secondary analysis of data that was prospectively collected from October 21 to November 11, 2013, and from February 1 to February 23, 2015, in the ED of a 700-bed urban Swiss academic tertiary care hospital with over 50,000 visits per year [4, 12]. The study protocol was approved by the local ethics committee (236/13, www.eknz.ch). The need to provide written informed consent was waived by the committee. The study was funded out of the research budget of the emergency medicine department at the University Hospital Basel.

All patients except obstetric, pediatric, and ophthalmologic patients are seen in the ED. The latter are treated in separate facilities nearby. We recruited patients presenting to the ED 24 hours a day, 7 days a week as part of a quality-control study. All patients presenting to the ED were eligible. The study team asked for verbal consent to participate. Patients were excluded if they actively declined participation or if the electronic health record contained a general rejection to participate in research. Patients who could not be interviewed or give consent because of dementia, intoxication, or severe language barriers were excluded. Patients who required active life-support, were unresponsive, or refused to participate were also excluded. A study team consisting of medical students who were unaware of the purpose of the study was instructed to systematically interview patients, nurses, and physicians. More details on the selection of participants and data collection can be found elsewhere [4].

Triage clinicians were asked the following question at the onset of each triage encounter: ‘How ill does this patient look?’ The disease severity was expressed on an 11-point numeric rating scale (NRS), from 0 (not sick at all) to 10 (extremely sick), with possible values 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10. We asked every patient at presentation whether they suffered from any of the following 35 symptoms: fever, skin rash, headache, dizziness, acute visual disorder, acute hearing disorder, nasal discharge, dysphagia, cough, expectoration, dyspnea, chest pain, abdominal pain, nausea, vomiting, diarrhea, constipation, dysuria, back pain, neck pain, arm pain, leg pain, joint pain, flank pain, joint swelling, leg swelling, altered state of mind, numbness, paralysis, gait disorder, speech disorder, fatigue, weakness, loss of appetite, sleeping disorder. Multiple answers were allowed. The Emergency Severity Index (ESI) score was defined as the urgency level at which a patient needed to be treated according to the German version of the ESI handbook. The Charlson Comorbidity Index (CCI) was calculated for each patient using the ICD-10 based Halfon-Version coding system coded by certified local coders.

Patient demographics (including age, gender and ethnic origin), the ESI triage category and the ICD-10 codes used to calculate the CCI were extracted from the hospital’s electronic health record. The symptoms at presentation including weakness and fatigue, were recorded at the time of inclusion in study case report forms based on systematic interviewing by the study team.

To blind the researchers, all predictors were assessed by trained medical students. Neither nurses nor physicians were aware of the study hypothesis when interviewed by the students immediately after performing the disease severity rating. Instead, they were informed that the reason of the assessment was a quality improvement project.

We investigated both short-term (acute morbidity, hospitalization, intensive care unit admission, in-hospital mortality) and one long-term outcome (1-year mortality), which were obtained from the electronic health record.

Acute morbidity was defined as any potentially life-threatening condition or any condition requiring early intervention to prevent disability, deterioration, or death [4, 13, 14]. It was assessed based on chart reviews by two physicians and, in case of disagreement, a senior physician who served as a referee. The results were documented on a data abstraction form. A patient was categorized as acutely morbid, if their chart included one or more of the following events: administration of antibiotics, virostatics, antifungals, immunosuppressives, diuretics, anticoagulants, antihypertensives, and procoagulants; the need for invasive interventions, or prolonged monitoring; new neurological deficits, or seizures; fractures, or self-harm. A more complete definition of acute morbidity can be found elsewhere [14].

In-hospital and 1-year mortality were defined as death before discharge from the hospital and death up to 365 days after presentation, respectively. Both were included because while in-hospital mortality is closer related to a patient’s acute medical challenges, 1-year mortality is linked to underlying chronic diseases. Patients were followed up for a year after presentation to assess 1-year mortality. A patients’ clinical condition as well as their emergency physicians’ rating thereof may not be causally related to the outcomes under study such as 1-year mortality. Nevertheless, if the disease severity rating can predict relevant outcomes, it could serve as a red flag for these patients, i.e., it could draw attention to patients who are likely to follow negative health and survival trajectories [15]. Reliable prognoses are an essential component of the practice of emergency medicine. For the individual patient, an unplanned emergency presentation can be a sentinel event with an impact on long-term prognosis [16]. In admitted patients, for example, it was shown that mortality rate was as high as 22% at 1 year. That study also found that admitted patients have an excess risk of dying compared to the background population [17]. Interestingly, although the odds ratio of death was highest in younger patients, the absolute risk of death was greatest in the elderly, suggesting that different disease trajectories come into play. Another study showed how the likelihood of death within the next 12 months is related to male gender, older age, admission to a medical specialty and social deprivation [18]. We have previously studied trajectories of illness in nonspecific complaints [15] and developed a framework containing five categories (functional, therapy-induced, deterioration of chronic condition, acute new condition, and acute event in a chronic condition). We found that each category has a distinct trajectory, as expressed by survival curves, thereby assisting prognostication. A related study found that certain factors, such as acute infections, were associated with unfavorable prognosis in patients with advanced dementia studied over the course of 2 years [19].

In summary, there is a strong need for hospitals to adopt a more evidence-based approach to identify patients who are entering the last year of their lives (i.e., 1 year follow-up). This prognostic information is needed in order to make and deliver appropriate care plans. Once entered into a patient’s electronic health record, the emergency physicians’ disease severity rating could be informative for a patient’s healthcare providers beyond their treating emergency physician.

Hospitalization was defined as admission to any hospital in-patient department, including disposition to other acute care hospitals directly from the emergency department within 24 hours. Intensive care unit admission was defined as admission to one of the hospital’s medical or surgical intensive care units, intermediate care units, or into stroke or neurosurgical intensive care during the index hospitalization (the hospitalization that we analyzed the data from).

The following outcomes were assessed in a blinded fashion using database matching: in-hospital mortality, 30-day mortality, hospitalization, and intensive care.

The analyses describing the population included the data of all patients. One-year mortality was missing for 227 patients. Therefore, these patients were excluded when analyzing 1-year mortality as an outcome. For the Receiver Operating Characteristics (ROC) curve analyses and the logistic regression models (see below), 49 patients were excluded because they had missing data on the disease severity rating (DSR; 43 missing) or the Emergency Severity Index (6 missing).

In all analyses patients with weakness and/or fatigue were compared with patients with neither weakness nor fatigue. To describe the patient population, patients were additionally split by gender for some analyses.

The predictive power of the three scores—disease severity rating (DSR), Emergency Severity Index (ESI), and, Charlson Comorbidity Index (CCI)—and their linear combinations were quantified using Receiver Operating Characteristics (ROC) curves [20], which plot for all possible thresholding values of a score the resulting sensitivity and specificity with respect to the outcome. The Area Under the ROC Curve (AUC) is a common metric to summarize the resolution (aka discrimination ability) of a score across all possible thresholding values. An AUC value can be interpreted as follows: If one patient with and one patient without an outcome (e.g., acute morbidity) is each randomly sampled from the corresponding group of patients, then the AUC value indicates the probability that the patient with the outcome has a higher score. In other words, AUC values represent the probability that a score correctly ranks patients according to their risk of experiencing an outcome.

We calculated AUC using the R package pROC v1.16.2 [21] using the trapezoid method (i.e., no smoothing); this method allows interpreting AUC values in terms of the probability of correctly ranking patients as discussed above. Furthermore, the direction of the scores (i.e., whether increasing or decreasing values indicate higher risk) was fixed (i.e., not estimated from the data) to avoid optimistic bias in the cross-validation analyses (see below).

When evaluating the three scores on their own, we calculated the AUC in-sample and report 95% confidence intervals (calculated using the DeLong method implemented in pROC). Since the score values are evaluated directly (i.e., no logistic regression models are used), there are no free parameters that are fit to the data; thus evaluating the performance in-sample is appropriate (i.e., no danger of overfitting).

When evaluating the performance of combinations of scores, we used standard logistic regression models (using the function glm with the family binomial(link = “logit”) in the stats R base package v4.0.2). To avoid overfitting the training data, we calculated AUC out-of-sample using 10-times repeated 5-fold cross-validation [22]. We report the mean AUC across validation folds and an approximate 95% confidence interval (calculated as mean AUC ±1.96·SD/√k, where SD is the sample standard deviation of the AUC values across the validation folds and k is the number of folds per CV repeat, here k = 5).

Sample size was not prospectively planned and was based on 5,634 presentations. In sum, 3,960 patients were included in the study (Fig 1).

Table 1 describes the study subjects’ characteristics, which are also reported elsewhere [4]. Patients presenting with W|F were more likely to be female (56%) than patients without these symptoms (47%), and they also had a higher median age of 55 years (IQR = 35–74) as compared to patients without W|F, whose median age was 49 (IQR 32–71). This age difference was more pronounced in male (58 years vs. 49 years) than in female patients (53 years vs. 50 years). While the median age in patients with neither weakness nor fatigue was similar in male (49, IQR = 32–69) and female patients (50, IQR = 33–73), females with W|F (median = 53, IQR = 35–73) were somewhat younger than males with either symptom (median = 58, IQR = 36–74). Male patients and female patients with W|F reported more symptoms than those without (median = 4 [IQR = 3–6] vs. 1 [IQR = 1–2] in males and median = 5 [IQR = 3–7] vs. 1 [IQR = 1–3] in females). In all four groups, the median ESI category was 3, with an IQR of 2–3 in patients with W|F and an IQR of 3–4 in patients without. In both males and females with W|F, the main other symptoms were dizziness and headache. Both genders with neither weakness nor fatigue most commonly reported leg pain and headache. Both genders with W|F were transferred to the ICU (males = 9% and females = 7%) more often than those without W|F (males = 6% and females = 4%). Based on their disease severity rating (Fig 2), ED physicians rated patients with weakness or fatigue as sicker (median = 4, IQR = 3–6) than all other patients (median = 3, IQR = 2–5).

Patients’ 1-year mortality was not recorded in 227 cases (5.7%). To investigate whether these patients’ characteristics differed from the rest, we compare the two groups in S1 Table. Patients with missing values were on average 14 years younger and were slightly more likely to have an ESI score of 4 or higher but did not differ from other patients in any noticeable way.

Table 2 shows the outcomes for patients with and without W|F, split by gender. Sixty-nine percent of all patients were treated as outpatients. The median length of stay (LOS) of inpatients was 5 days. Forty percent of all patients suffered from acute morbidity, 6% were transferred to the ICU, 1% died in hospital, and 5% died within a year. One-year follow-up was completed for 3,733 patients (94%). Patients with W|F (n = 733) differed from all other patients: Acute morbidity was found in 45% of patients with W|F and in 38% of all other patients. One-year mortality was 8% in patients with W|F and 5% in all other patients. In-hospital mortality was 2% in patients with W|F and 1% in all other patients. Hospitalization rate was 42% in patients with W|F and 29% in all other patients. Intensive care use was 8% in patients with W|F and 5% in all other patients.

Figs 3 and 4 show the extent to which DSR, the Emergency Severity Index (ESI), and the Charlson Comorbidity Index (CCI) can predict the five outcomes, separately for patients with and without weakness and/or fatigue (W|F). Fig 3 shows Receiver Operating Characteristics (ROC) curves and Fig 4 shows the Areas Under those ROC curves (AUC; Fig 4). Results show that DSR can predict all five outcomes for both patient groups well above chance (i.e., AUCs > ~0.70). Furthermore, there is no clear evidence that DSR performs worse for patients with W|F.

Figs 3 and 4 show that DSR predicts patient outcomes as good as or better than ESI and CCI. The same pattern holds also when separately comparing the scores for patients with and without weakness and/or fatigue (W|F) with one exception: There is some evidence that for patients with W|F CCI predicts 1-year mortality better than DSR.

The above analyses did not adjust for typical covariates, such as age and sex. For a complete picture, we contrast the predictive power of DSR vis-á-vis age and sex to gauge DSR’s predictive power beyond those two standard demographic variables. S1 Fig shows how DSR and age relate to each other, separately for female and male patients with or without symptoms of weakness or fatigue. The figure shows that DSR increases with older age, but there is clear excess variance not explained by age. Additionally, to check whether DSR predicts the outcomes beyond a control model that accounts for age and sex, we compared two additional models (age + sex, DSR + age + sex) to the DSR-only model and report this comparison in S2 Fig. Results show that DSR clearly provides non-redundant predictive information beyond age and sex for acute morbidity, hospitalization, and transfer to ICU (i.e., the DSR + age + sex model outperforms the age + sex model). In contrast, for 1-year mortality and in-hospital mortality the combined model (DSR + age + sex) does not clearly perform better than the control model (age + sex).

Fig 5 shows the extent to which disease severity rating (DSR), the Emergency Severity Index (ESI), the Charlson Comorbidity Index (CCI), and all linear combinations of two or three scores can predict the five outcomes, separately for patients with and without weakness and/or fatigue (W|F). Fig 6 shows absolute changes in the AUC when adding DSR to ESI, CCI, or a linear combination of ESI and CCI. For acute morbidity, hospitalization, and transfer to ICU there is clear evidence that DSR improves predictions for both patients with and without W|F; for 1-year mortality and in-hospital mortality this result holds for most, but not all comparisons. Consistent with the above-reported result that CCI performs better than DSR in predicting 1-year mortality in patients with W|F, adding DSR to CCI does not clearly improve performance. Because of the low prevalence of in-hospital mortality in the current population (1–3%; see Table 2), the results for this outcome come with considerable uncertainty. Having said that, there is some evidence that DSR improves on ESI, but not on CCI or a combination of ESI and CCI.

Fig 7 shows absolute changes in the AUC when adding the Emergency Severity Index (ESI), the Charlson Comorbidity Index (CCI), or their linear combination to DSR. The results show that generally adding one or both benchmark scores to DSR does not improve performance or even decrease it, with two exceptions. First, consistent with the above-reported results on 1-year mortality, adding CCI to DSR does improve performance for patients with weakness and/or fatigue (W|F). Second, for hospitalization, adding ESI and CCI to DSR improves performance for both patient groups; however, only adding ESI or CCI to DSR does not.