Let the incidence or number of newly infected cases in an epidemic at present time t, be I_t. The incidence curve is a historical record of these case counts from the start of the outbreak and is summarised as I 1 t = { I s : 1 ≤ s ≤ t }, with s as a time-indexing variable. For convenience, we assume that incidence is available on a daily scale so that I 1 t is a vector of t daily counts (weeks or months could be used instead). The associated effective reproduction number and total infectiousness of the epidemic are denoted R_t and Λ_t, respectively.

Here R_t is the number of secondary cases that are induced, on average, by a single primary case at t [1], while Λ_t measures the cumulative impact of past cases, from the epidemic origin at s = 1, on the present. The generation time distribution of the epidemic, which describes the elapsed time between a primary and secondary case, controls Λ_t (see the S1 Text). In the problems we consider, I 1 t and Λ 1 t form our data, the generation time distribution is assumed known, and R 1 t or groupings of this vector are the parameters to be inferred.

The renewal model [5] derives from the classic Euler-Lotka equation [6] (see the S1 Text), and defines the Poisson distributed (Poiss) relationship I_t ∼ Poiss(R_tΛ_t). This means that ℙ(x)=1x!e−RtΛt(RtΛt)x, with x indexing possible I_t values. This formulation assumes maximum epidemic non-stationarity (i.e. that demographic transmission statistics change at every time unit) and hence only uses the most recent data (I_t, Λ_t) to infer the current R_t. While this maximises the fitting flexibility of the renewal model, it often results in noisy and unreliable estimates that possess many spurious reproduction number changes [7]. Consequently, grouping is employed.

This hypothesises that the reproduction number is constant over a k-day window into the past and results in a piecewise-constant function that separates salient fluctuations (change-points) from negligible ones (the constant segments) [10, 11]. We use R_τ(t) to indicate that the present reproduction number to be inferred is stationary (constant) over the last k points of the incidence curve i.e. the time window τ(t) ≔ {t, t − 1, …, t − k + 1}. The data used to estimate R_τ(t) is then ( I t - k + 1 t , Λ t - k + 1 t ). This construction allows us to filter noise and increase estimate reliability but elevates bias. In the S1 Text we show precisely how grouping achieves this bias-variance trade-off.

Choosing k therefore amounts to selecting a belief about the scale over which the epidemic statistics are meaningfully varying and can significantly influence our understanding of the population dynamics of the outbreak. Thus, it is necessary to find a principled method for balancing k. Ultimately, we want to find a k, denoted k*, that (i) is best supported by the epi-curve and (ii) maximises our confidence in making real-time, short-term predictions that can inform prospective epidemic responses. This is no trivial task as k* would be sensitive to both the specifically observed stochastic incidence of an epidemic and to how past infections propagate forward in time.

We solve (i)-(ii) by applying the accumulated prediction error (APE) metric from information theory [13], which values models on their capacity to predict unseen data from the generating process instead of their ability to fit existing data [14]. The properties and mathematical definition of the APE are provided in the S1 Text and in the subsequent section. The APE uses the window of data preceding time s, ( I s - k + 1 s , Λ s - k + 1 s ), to predict the incidence at s + 1 and assigns a log-score to this prediction. This procedure is repeated over s ≤ t and for possible k-values. The k achieving the minimum cumulated log-score is deemed optimal. Fig 1 summarises and illustrates the APE algorithm.

To adapt APE, we require the posterior predictive incidence distribution of the renewal model, which at time s is P ( x | I s - k + 1 s ), with x indexing the space of possible one-step-ahead predictions at s + 1, and I ^ s + 1 = E [ x ]. We assume a gamma conjugate prior distribution on R_τ(s) as is commonly done in the renewal model frameworks of [3, 10] and because the gamma distribution can fit many unimodal variables on the positive real line. For some hyperparameters a and c this is R τ ( s ) ∼ Gam ( a , 1c ), with Gam as a shape-scale parametrised gamma distribution. The posterior distribution of R_τ(s), P ( R τ ( s ) | I s - k + 1 s ), is then R τ ( s ) | I s - k + 1 s ∼ Gam ( a + i τ ( s ) , 1 c + λ τ ( s ) ) . (1)

For convenience we define α τ ( s ) ≔ a + i τ ( s ) , β τ ( s ) ≔ 1c + λ τ ( s ) with i_τ(s) and λ_τ(s) are the sum of incidence (I_s) and total infectiousness (Λ_s) over the window τ(s) (see the S1 Text). If a variable y ∼ Gam(α, β) then ℙ(y)=yα−1e−y/β/βαΓ(α) and E [ y ] = α β. The posterior mean estimate is therefore R ^ τ ( t ) = α τ ( t ) β τ ( t ). Applying Bayes formula and marginalising yields the posterior predictive distribution of the number of infecteds at s + 1 as P ( x | I s - k + 1 s ) = ∫ 0 ∞ P ( x | R τ ( s ) ) P ( R τ ( s ) | I s - k + 1 s ) d R τ ( s ) . (2)

In Eq (2) we used the conditional independence of future incidence data from the past epi-curve, given the reproduction number to reduce P ( x | I s - k + 1 s , R τ ( s ) ) to P ( x | R τ ( s ) ), which expresses the renewal model relation x ∼ Poiss(R_τ(s)Λ_s+1). As Λ_s+1 only depends on I s - k + 1 s there are no unknowns. Solving using this and Eq (1) gives P ( x | I s - k + 1 s ) = ϕ 1 ϕ 2 - 1 where ϕ 1 ≔ Λ s + 1 x ∫ 0 ∞ R τ ( s ) x + α τ ( s ) - 1 e - R τ ( s ) ( Λ s + 1 + 1 β τ ( s ) ) d R τ ( s ) and ϕ 2 ≔ β τ ( s ) α τ ( s ) Γ ( x + 1 ) Γ ( α τ ( s ) ) , with Γ ( z ) ≔ ∫ 0 ∞ θ z - 1 e - z d θ . (3)

Since ∫θz−1e−yθdθ≡Γ(z)yz the integral in ϕ₁ simplifies to Γ(x+ατ(s))(Λs+1+1βτ(s))x+ατ(s). Some algebra then reveals the negative binomial (NB) distribution: x | I s - k + 1 s ∼ NB ( α τ ( s ) , p τ ( s ) ≔ Λ s + 1 β τ ( s ) 1 + Λ s + 1 β τ ( s ) ) . (4)

If some variable y ∼ NB(α, p) then P ( y ) ≔ ( α + y - 1 y ) ( 1 - p ) α p y and E[ y ]=pα1−p. Eq (4) is a key result and relates to a framework developed in [3]. It completely describes the one-step-ahead prediction uncertainty and has mean E [ x ] = I ^ s + 1 = Λ s + 1 R ^ τ ( s ), and variance var ( x ) = I ^ s + 1 ( 1 + I ^ s + 1 i τ ( s ) + a ). These relations explicate how the current estimate of R_τ(s) influences our ability to predict upcoming incidence points. At s = t the above expressions yield prediction statistics for the next (unobserved) time-point beyond the present.

The APE metric is generally defined as APEk ≔ ∑ s = 1 t - 1 - log P ( I s + 1 | I s - k + 1 s ) [13] (see the S1 Text) and is controlled by the shape of the posterior predictive distribution over time. We specialise this metric for renewal models and integrate Eq (4) into the algorithm of Fig 1 to derive APE k = ∑ s = 1 t - 1 B s + 1 + I s + 1 log p τ ( s ) + α τ ( s ) log ( 1 - p τ ( s ) ) , (5) with I_s+1 as the (true) observed incidence at time s + 1, which is evaluated within the context of the predictive space of x, and B s + 1 ≔ log ( I s + 1 + α τ ( s ) - 1 I s + 1 ). Eq (5) is exact, easy to evaluate and offers a simple and direct means of finding the optimal data-justified window length as k* ≔ arg min_kAPE_k.

The summation in Eq (5) also clarifies why our approach is useful for real-time applications. As an epidemic unfolds and more data accumulate the upper limit on s will increase. We can update APE_k to account for emerging data by simply including an additional term for each new data point on top of the previously calculated sum. For example, a new datum at t + 1 requires the addition of B_t+1 + I_t+1 log p_τ(t) + α_τ(t) log (1 − p_τ(t)) to Eq (5). In the Results section we will demonstrate this by successively computing k s *, the optimal window length for data up to time s.

APE_k can also be computed using the in-built NB routines of many software (some parametrise this distribution differently so that Eq (4) may need to be implemented as NB(α_τ(s), 1 − p_τ(s))). When computing Eq (5) directly, the most difficult term is B_s+1 when I_s+1 and α_τ(s) are large. In these cases Stirling approximations may be applied. We provide Matlab and R implementations of our renewal model APE metric at https://github.com/kpzoo/model-selection-for-epidemic-renewal-models.

We apply our APE metric to select k* for several epidemic examples, which examine reproduction number profiles featuring stable (Fig 2A) and seasonally fluctuating (Fig 2B) transmission, as well as exponential (Fig 3A) and step-changing (Fig 3B)) control measures. These examples explore dynamically diverse epidemic scenarios and consider both large (Figs 2A and 3A) and small outbreaks (Figs 2B and 3B). Small outbreaks are fundamentally more difficult to estimate. We simulated epidemics for t = 200 days using a generation time distribution analogous to that used for Ebola virus disease predictions in [15]. We investigated a window search space of 2≤k≤t2 and computed the APE at each k, over the epidemic duration (1 ≤ s ≤ t). Reproduction number estimates, R ^ τ ( s ), and demographic incidence predictions x | I s - k + 1 s, are presented in Figs 2 and 3. Predictions over the first s < k times use all s data points.

We find that the APE metric balances R ^ τ ( s ) estimate accuracy against one-step-ahead predictive coverage (i.e. the proportion of time that the true I_s+1 lies within the 95% prediction intervals of x | I s - k + 1 s) for a range of R_s dynamics (top graphs of Figs 2 and 3). This dual optimisation is central to this work. Moreover, vastly different I_s+1 predictions and R ^ τ ( s ) estimates can result when k is misspecified (middle and bottom graphs). This can be especially misleading when attempting to identify the significant changes in epidemic transmissibility and can support substantially different beliefs about the infection population biology. Estimation and prediction performance also depend on the actual incidence at any time; small I_s values lead to wider confidence intervals in R ^ τ ( s ) at any k (see Eq. (S3) of the S1 Text).

When k is unjustifiably large not only do we observe systematic prediction and estimation errors, but alarmingly, we tend to be overconfident in them. When k is too small, we infer rapidly and randomly fluctuating R ^ τ ( s ) values, which sometimes can deceptively underlie reasonably looking incidence predictions. Consequently, optimal k-selection is integral for trustworthy inference and prediction. Observe that small k, which implies a more complex renewal model (i.e. there are more parameters to be inferred), does not generally result in better causally predictive one-step-ahead incidence predictions. Had we instead naively picked the k that best fits the existing epi-curve, then the smallest k would always be favoured (overfitting) [11].

We emphasize and validate the predictive performance of APE in Fig 4. This shows, for all simulated examples above that minimising the APE also approximately minimises the percentage of the time s ≤ t that true incidence values fall outside the 95% prediction intervals of Eq (4). This validates the APE approach and evidences its proficiency at optimising for short-term forecasting accuracy in real time. We recommend using APE to define k* for an existing epi-curve up to the present t. Applying Eq (4) with this k* should then best predict the number of cases on the (t + 1)^th day. This entire procedure should be repeated for later forecasts, with k* being progressively updated. The real-time behaviour of k* as data accumulate over the course of an outbreak is investigated in the next section.

During an unfolding epidemic, renewal models can support response efforts by providing real-time estimates of infectious disease transmissibility and short-term forecasts of the expected incidence of cases [9]. Here we show how the APE metric can be used within this framework as a rapid and reliable real-time tool. We focus on the key problem of diagnosing and verifying the efficacy of implemented control measures over the course of an emerging outbreak [3]. Knowing whether an intervention is working or not can inform preparedness and resource allocation. Estimating whether R_s < 1 or R_s ≥ 1 is one simple means of assessing efficacy. Effective control measures lead to the former.

We investigate how the APE metric responds in real time to swift swings in reproduction number and quantify our results over 10³ simulated epi-curves with t = 150 days. We focus on step-changes in R_s and examine how the successively optimal window length, denoted k s * when computed with data up to time s, responds. If k s * is sensitive to these changes with minimum delay then it is likely a dependable means of diagnosing control efficacy in real time. In previous sections we have been computing k * = k t *. We consider four models and present our main results in Figs 5 and 6.

In Fig 5 we examine cases of increasing transmission for (A) an outbreak that is uncontrolled and worsening (e.g. if the infectious disease acquires an additional route of spread) and (B) an ineffectively controlled epidemic with a late-stage elevation in R_s (e.g. if quarantine measures were relaxed too quickly). Fig 6 then investigates scenarios involving effective interventions where (A) rapid control is employed (e.g. if a wide-scale lockdown is enacted) and (B) the initial action is only partially effective and so later additional controls are needed (e.g. if flight restrictions were first applied but then further transport shutdowns or mobility restrictions were required).

In both figures, top graphs overlay true I_s+1 and predicted x | I s + k - 1 s across time and middle ones show the best R ^ τ ( s ) estimates under the final k t *. We see that the APE-selected model properly distinguishes significant changes from stable periods in all scenarios. The model in Fig 5A is the most difficult to infer as the increasing R_s does not visibly change the shape of the incidence population curve. Bottom graphs show k s * sequentially across the ongoing epidemic. Here we observe how the APE metric uses data to justify its choices in real time. As data accumulate under a stable reproduction number APE increases k*. This makes sense as there is increasing support for stationary epidemic behaviour. This is especially obvious in Fig 6A.

However, on facing a step-change the APE immediately responds by drastically reducing k s * − both in scenarios where R_s is changing from above to below 1 and vice versa. This reduction is less visible in Fig 5A because the observed epi-curve is not as dramatically altered as in the other scenarios. This rapid response recommends the APE as a suitable and sensitive real-time diagnostic tool. We dissect the impact of change-times on the actual APE values at different k in Fig 7. There we find that when the epi-curve is reasonably stable there is not much difference between the performance at various window lengths and so the APE curves are neighbouring. In contrast, when a non-stationary change occurs there is a clear unravelling of the curves and potentially large gains to be made by optimising k. Failing to properly select k here would potentially mislead our understanding of the unfolding outbreak. This increases the impetus for formal k-selection metrics such as the APE.

We test our APE approach on two well-studied empirical epidemic datasets: pandemic H1N1 influenza in Baltimore from 1918 [16] and SARS in Hong Kong from 2003 [17]. For each epidemic we extract the incidence curve, total infectiousness and generation time distributions from the EpiEstim R package [10] and apply the APE metric over 2≤k≤t2 with t as the last available incidence time point. We compare the one-step-ahead I_s+1 prediction fidelity and the R_τ(s) estimation accuracy obtained from the renewal model under the APE-selected k* to that from the model used in [10], which recommended weekly windows (i.e. k = 7) after visually examining several window lengths.

Our main results are in Figs 8–10. We benchmarked our estimates against those directly provided by EpiEstim to confirm our implementation and restrict k > 1 to avoid model identifiability issues [11, 18]. Intriguingly, we find k* = min k = 2 for both datasets. This yields appreciably improved prediction fidelity (i.e. the coverage of observed incidence values by the 95% prediction intervals), relative to the weekly window choice, as seen in Fig 8. Between 8−10% of incidence data points are better covered by using k* over k = 7. This improvement is apparent in the right graphs of Figs 9A and 10A, where the k = 7 case produces stiffer incidence predictions that cannot properly reproduce the observed epi-curve.

Weekly windows misjudge the SARS epidemic peak, predict a multimodal SARS incidence curve that is not reflected by the actual data and systematically underestimate influenza case counts when it matters most (i.e. around the high incidence phase). However, the smaller k* window results in noisier R_τ(s) estimates (left graphs of Figs 9A and 10A, which may be questionable. These rapidly fluctuating reproduction numbers likely motivated the adoption of weekly windows in previous analyses. While the APE-based estimates are indeed more uncertain (a consequence of shorter windows), we argue that they are formally justified by the available data, especially given the weaker predictive capacity of weekly windows. While the noisier R_τ(s) estimates do not mislead our understanding of the efficacy of implemented control measures (it is still clear that the influenza epidemic is only partially under control between 40 ≤ s ≤ 65 days before recovering, while the SARS outbreak is largely arrested from s > 50 days), they likely overestimate the peak transmissibility of these diseases.

We might suspect that certain artefacts of the data could resolve this issue, rendering a more believable combination of estimated reproduction number and predicted incidence. Particularly, the influenza data seem considerably more affected by the smaller look-back windows. These k* = 2 windows are needed to help predictions get close to the peak incidence values of the data. However, these peaks seem reminiscent of outliers and in the original analysis of [16] they were attributed to possible recollection bias in patients that were questioned. Removing these biases might be expected to lead to smoother APE-justified reproduction numbers. We test this hypothesis in Figs 9B and 10B.

There we find that applying simple 5-day moving average filters to the data, as done in [16] to ameliorate outliers, still does not support a k* > 2 in either scenario, though smoother estimates do result. Weekly windows are still too inflexible to properly predict this averaged incidence. While other signal processing techniques, such as using autocorrelated smoothing prior distributions instead of independent gamma ones over the reproduction numbers, could be applied to further investigate if k* > 2 is justifiable we consider this beyond the scope of this work and somewhat biologically unmotivated. Instead we conjecture that these results support the interesting alternative hypothesis that the epi-curve population data are not Poisson distributed. We expand on this conjecture in the subsequent section.