We employ the publicly available virsim model [6], which is a stylized representation of the COVID-19 epidemic with geographical stratification of both transmission and interventions (i.e., a meta-population model). More specifically, it is an agent-based model with a geographical structure defined by means of clusters (representing for instance towns and villages) and superclusters (e.g. provinces or other administrative units for which policy decisions are made). Individuals are separated into susceptible (S), exposed (E), infectious (I) and recovered (R), and life-long immunity is assumed. The addition of geographical stratification allows for heterogeneity at the individual and sub-popolation level, as well as the implementation and evaluation of regional approaches. A technical description of the model is provided in S1 Text and S1 Table. For further details on the model we refer the reader to de Vlas & Coffeng [5]. For this study, we employed an updated quantification of the model [9]; see Table 1 in the main text (with the corresponding distributions displayed in S1 Fig) and S1 Table for an overview of the parameter values.

We consider four exit strategies: Flattening the Curve (FC), Contact Tracing (CT), Intermittent Lockdown (IL) and Phased Opening (PO). Each strategy is part of the model implementation (see S2 Text for the computational details) and is defined by a unique set of parameters. More details about the strategies parameters are given in a later section, where we discuss also the input distributions that have been considered to account for the respective uncertainties.

Since this study is of conceptual nature and does not aim to model real-world scenarios as accurately as possible, we simulate a population of 1 million individuals and we focus on the first year after the implementation of a strategy (the idea being that governments will regularly evaluate and adapt the strategy in use). In the Discussion section we consider what is needed to bring our analysis to more complex models (and strategy implementations) that aim to be more realistic. All simulated intervention strategies are preceded by a period of lockdown, analogous to the situation in the Netherlands from March to May 2020. During this period, we assume that an intervention package is implemented that heavily reduces transmission. The model parameter for this effect (intervention_effect) is expressed in terms of the level of transmission that the intervention package still allows, relative to a situation without any interventions (i.e., 100% minus the reduction in the overall transmission rate). Based on a previous quantification of the model using extensive data on the Dutch COVID-19 outbreak [9], we assume that the lockdown reduces transmission to 30% on average for one week and then to 15% on average after the Dutch government introduced stricter measures. In accordance with the Dutch situation, the effect of the lockdown is simulated for 60 days, followed by a period of partial lockdown (assumption: intervention_effect = 25%), analogous to the re-opening of schools in the Netherlands [9]. After another 30 days, one of the exit strategies takes effect in the simulation. Overall we simulate a period of 550 days corresponding to roughly 1.5 years; see S2 Text for details about how this is implemented in the model.

Uncertainties in model output can arise from different sources; we discuss here four main types. The first is parameter uncertainty, referring to uncertainties in model parameters whose values can be set directly by the model user via the inputs of the computational model. An example is the reduction of the transmission rate due to the introduction of an intervention. For the COVID-19 epidemic, a significant decrease in the transmission rate following the adoption of such measures has been recorded [10–13], however the magnitude of this reduction is uncertain.

The second type of uncertainty, which we call intrinsic uncertainty, arises when a computational model is inherently stochastic. In epidemiology, many models are agent-based and possess internal stochasticity, for instance in the randomized interactions between agents. Model users often have little control over such internal stochasticity as they can typically only set the seed of the random number generator at the start of a simulation [14].

The third type is model-form uncertainty, referring to uncertainty or errors in the structure of the model itself (e.g. due to transmission mechanisms not represented in the model). This type of uncertainty cannot be analyzed by changes in the model inputs but requires a comparison either with independent observation data or with other models—as done for example in climate science [15].

Lastly, initial condition uncertainty is due to the inaccuracies in the specification of the initial state of the model (i.e., the state of the simulated population at the start of the model run). Since we consider here model outputs which are independent from the specific timing of e.g. epidemic peaks, this type of uncertainty is not important in our analysis.

In this study we analyze the parameter and intrinsic uncertainties by means of non-intrusive UQ methods, which means that we treat the computational model (i.e., the virsim model) as a black box. We extend the notation from the introduction to Y=M(X,r), where r denotes the seed of the random number generator. By random sampling of the parameter vector X from its distribution p(X) and altering the seed r, followed by executing the model M, we create random samples from the probability distribution of Y. Thus, we can probe the uncertainty of Y and estimate its distribution by repeated model executions.

In order to assess which parameters create the most uncertainty in the model output Y under the different strategies, we compute the Sobol indices [16]. This is a form of global sensitivity analysis focusing on the variance of Y. Assuming mutual independence among the input parameters, Var(Y) is decomposed into fractions which can be attributed either to a single input parameter (first order Sobol indices) or to a set of parameters (higher order Sobol indices). Given p(X), the first order Sobol index for the i-th input parameter (X_i) is defined as Si≔Vari[E∼i(Y|Xi)]/Var(Y), as explained in S3 Text. We use the notation Ei for expectation over X_i and E∼i for expectation over all X₁, X₂, … except X_i (and similarly for Var_i and Var_∼i). If S_i is close to 1, it means that the variance of Y is almost entirely due to the variance of X_i. The overall effect on the model output of all parameter combinations involving X_i is given by the total Sobol index STi≔1-Var∼i[Ei(Y|X∼i)]/Var(Y). We remark that the Sobol index for the seed of the pseudo-random number generator can be non-negligible. It quantifies the contribution of the internal stochasticity to the overall variance of the output, i.e. the QoIs.

Regarding the assumed mutual independence of the input parameters, there might be dependencies when parameters are actually estimated from data. However such dependencies are not implemented in the computational model, hence the selected input parameters (see below for more details) are effectively treated as mutually independent. We refer to S3 Text for more details about the theory of the Sobol indices.

In our analysis we consider the model predictions for the number of incident and prevalent individuals in the population that require IC admission. As IC capacity is limited, the question whether the IC capacity will be exceeded (and if so, by how much) according to model predictions is clearly important. To investigate this, for each simulation we consider two quantities of interest (QoI):

the maximum of the moving average of the prevalent cases in intensive care (averaging window = 30 days)

The first QoI shows the peak value of the number of COVID-19 patients in IC units, giving an indication of the intensity of an outbreak. We apply a moving average to focus on longer-term trends, filtering out short-term “noisy” variations. When analyzing robustness, a natural threshold for this QoI is the available IC capacity, which may vary from country to country and from month to month. In our analysis we assume the IC capacity to be constant in time and we consider the Netherlands as reference country. De Vlas and Coffeng [5] calculate that during the first epidemic wave in 2020 the Dutch maximum IC capacity for COVID-19 patients was 109 IC beds per million inhabitants.

The second QoI quantifies by how much the total IC capacity is overburdened. It is defined as ∑t=0Tmax(0,p(t)-p*) where p(t) is the number of IC prevalent cases at day t ∈ {0, 1, 2, …, T}, p* is the maximum IC capacity and the summation runs from the start time of the simulation (t = 0 day) to the final time (t = T days). It shows the extent to which the government may have to rely on other countries, e.g. Germany, to take in IC patients from the Netherlands. Setting a threshold for this QoI is complicated, as many economical and political factors come into play here.

We note that the two QoIs as defined here do not depend on time. The first QoI is defined as a maximum over the time interval of simulation, whereas the second QoI is defined as a summation over the same time interval. Thus, a single model simulation yields a single, time-independent value for each QoI.

The implementation of a strategy in the computational model is determined by a set of input parameters for the model. Some of the parameters can be controlled (up to a certain degree) by policy makers. These are parameters related to social aspects of the population—e.g. social distancing—and to the availability of resources, e.g. to test and track infected individuals. They can be controlled to some extent by government authorities through imposing stricter (or less strict) rules for e.g. social distancing, or through making more (or less) resources available. We call them policy-related parameters and we refer to the uncertainties generated by these parameters as policy-related uncertainties.

Below we provide details about which policy-related parameters are treated as uncertain, together with the rationale behind the probability distributions chosen for them. The distributions, together with their mean and 95%-confidence intervals are provided in Table 1; their plot instead is provided in S1 Fig.

We use Beta distributions for those parameters that are naturally restricted to values on a finite interval, such as those representing probabilities or percentages. We remark that this allows us to use distributions that are very different from the uniform distribution. With the hyperparameters used here, the probability density of the Beta distribution decreases to zero towards the boundaries of the support. By contrast, the uniform distribution attributes equal probability to any value within the support of the distribution. For parameters defined as a time scale or a rate we choose the Γ distribution as it has one unique peak (with the hyperparameters used here) and a semi-infinite domain (all non-negative values).

Our choice of the distributions was guided by expert knowledge. Ideally, as more data become available the initial guesses for those parameters that can be directly estimated could be updated to better reflect the data (although this has not been done here). However, this is not always possible as there may be parameters that cannot be estimated directly from data. Furthermore, parameters can be model specific. Relatedly, some of our selected distributions are not transferable to other transmission models if these have explicit representations (with dedicated parameters) of processes only implicitly represented in virsim. For instance, in the virsim model the parameter for the effect of the intervention in Flattening the Curve implicitly includes the effect of different measures (e.g. social distancing, working from home and closure of schools), therefore it cannot be directly estimated from data for every plausible combination of interventions. Hence it is advisable to combine expert knowledge and calibration on the available data to determine the most appropriate input distributions.

Besides the parameters that can be influenced by policy, discussed in the previous section, the computational model has other parameters that we include in our analysis. Examples are the reproduction number of the virus and the duration of infectiousness. Because these parameters have to be estimated from data, their values can be rather uncertain, especially when available data is scarce (for instance in the early stages of an epidemic with a new virus). For some parameters, care has to be taken when sampling them from probability distributions, since certain basic characteristics, like the doubling time, have to match the time evolution of the epidemic as captured in the (real-world) observation data. We chose distributions whose mean is (almost) the value obtained in a previous quantification of the model [9]. Using Γ distributions was a modelling decision and the overall shape of the distributions was guided by expert knowledge.

Altogether, given the setup of the virsim model, we consider the following non-policy-related uncertain parameters:

the average duration of infectiousness (model parameter avg_duration_infectiousness) has been estimated to be about 5 days. To account for errors and a conceivable lack of data in the estimation process, we sample it from a Γ distribution;

The virsim model is stochastic, and as discussed in an earlier section, this internal stochasticity is a source of uncertainty of the model output. It is not possible to choose the probability distribution of the internal (or latent/hidden) random variables of the model, or to set them by hand to specific values according to some sampling plan. The only form of control as model user is to pick the seed for the pseudo-random number generator at the start of the simulation. An example of uncertainties in the model output due to the internal stochasticity is provided in S2 Fig for the FC strategy with fixed parameters but varying seed. It can be seen that the peak value of the number of prevalent cases in IC for different realizations can vary between 150 and 200 leading to differences up to 25% among the simulations. This shows that the model has significant internal stochasticity. We draw the seed from a discrete uniform distribution with support between 16384 and 65536. This amounts to independent random sampling of the internal random variables of the virsim model.

Furthermore, when using models with a geographical structure like the virsim model, uncertainties can arise from the level of geographical mixing that is allowed in the model. For sake of simplicity we do not consider such uncertainties in the present study, but they should be considered in more comprehensive studies if these are intended to inform policy makers.

We report in Table 1 the assumed input distributions, their mean and 95%-confidence intervals reflecting parameter uncertainty for model parameters that were considered in the UQ analysis. The plot of the distributions themselves is provided in S1 Fig. Finally, in S1 Table we list the model parameters that have fixed values (i.e., the parameters not considered as uncertain) in this study. The reported values have been taken from the model quantification used in [9].

If the dependence of the QoIs on the parameters is smooth and the number of uncertain parameters is not too high, the propagation of uncertainties from parameters to QoIs can be assessed with techniques such as Polynomial Chaos Expansion and Stochastic Collocation [7, 8, 17], that allow to obtain the relevant information with relatively little computational time. However, due to the presence of intrinsic uncertainty, the continuous relation between QoIs and inputs is lost as the QoIs do not depend continuously on the random seed. When this type of uncertainties is present, the aforementioned methods like Polynomial Chaos Expansion are not suitable. Therefore, we resort to Monte Carlo (MC) sampling as it is guaranteed to return reliable results (albeit with a low convergence rate). We base our UQ results on the evaluation of an MC ensemble with 2000 simulations in total. MC sampling implies that for each simulation, the full vector of uncertain parameters is sampled anew (randomly). More advanced techniques meant to overcome the issue of the internal stochasticity are currently being developed, see for instance [18] but require further investigation before application to high-dimensional systems.

For the SA, we compute the first order Sobol indices with the cost effective algorithm of Saltelli [19] (see S3 Text), and their 95%-confidence intervals are computed via bootstrapping. By performing an intelligent sampling of the input parameters, this algorithm reduces the computational cost of the calculation of the first order Sobol indices from M²p to M(p + 2) model simulations, where M is the number of MC samples and p is the number of uncertain parameters (including the random seed). For more details about the practical implementation of the Saltelli algorithm please see [8]. Due to the specific sampling required by the Saltelli algorithm, the SA cannot be performed on the same set of data employed for UQ. Therefore we run a separate set of simulations, and use M = 2000 MC samples per parameter involved in the implementation of the strategy (SA is in general computationally more demanding than UQ). This results, e.g., in case of FC in a total of M(p + 2) = 2000(3 + 2) = 10000 simulations, where we considered only the random seed and the policy-related parameters.

Sampling and post-processing analysis are done using the Monte Carlo sampler of the publicly available Python library EasyVVUQ [20–22]. We run the required model simulations in parallel on a supercomputer at the Poznan Supercomputing and Networking Center. For the job submission to the supercomputer we use the FabSim3 [23, 24] and QCG-PilotJob [25] packages. FabSim3, QCG-PilotJob and EasyVVUQ are all part of the open source VECMA Toolkit (VECMAtk) [26, 27]. The codes employed for this study can be found on GitHub [28].

As first step of our UQ analysis, we construct the empirical cumulative distribution functions (cdfs) of our QoIs. For any threshold value q*, the cdf gives the probability that the QoI remains below (or at) that threshold, i.e. it gives P(QoI≤q*) for the adopted strategy (and given the specific input parameter ditributions). In Fig 2 we report the resulting cdfs for the four selected strategies both with and without uncertainties in non-policy-related parameters (in the latter case, we fix these parameters at the mean values listed in Table 1, except for shape_exposed_time which is set to 20, see also S1 Table, and for the seed which is never fixed). We further display the 95%-confidence interval of the empirical cdfs using the Dvoretzky-Kiefer-Wolfowitz inequality [29, 30], which gives an indication on how reliable the empirical cdfs are.

We observe the important fact that, with the given distributions of the input parameters, none of the analyzed strategies is robust. The probability that the number of prevalent patients in intensive care is larger than the IC capacity is rather high and only Contact Tracing gets close to a probability of 50%. This shows that the assumed input distributions for Flattening the Curve, Intermittent Lockdown and Phased Opening correspond to interventions that are not sufficiently restrictive to stay below the threshold, as far as the model is concerned.

We note that the cdfs of the first QoI for CT and IL increase more gradually compared to FC and PO (implying that the variance of the first QoI is larger for CT and IL than it is for FC and PO). Furthermore, it can be seen that the shape of the cdfs is only weakly affected by non-policy-related parameters. Therefore, when searching for the parameters responsible for most of the output variability, i.e. for the sensitivity analysis, these parameters might be kept fixed to reduce the computational burden. They should be included however when determining the minimum level of e.g. intervention or uptake, required for the QoIs to stay below their threshold with 95% probability.

In Fig 3 we report, for the first QoI, the 95%-confidence interval of the first order Sobol indices for the policy-related parameters and for the seed. This information allows us to identify which factors of the strategies most affect the model output, such that measures targeting these factors can be adopted in order to make the strategy robust.

From the width of the intervals, it can be seen that the number of MC samples (M = 2000) is too small to estimate the Sobol indices very accurately. Nonetheless a qualitative ranking of the parameters with respect to the generated output variance can be made, based on the results in Fig 3. It is therefore possible to distinguish intervention_effect, trace_rate_I, uptake and pl_intervention_effect_hi as the parameters responsible for most of the output uncertainty, and should thus be targeted (e.g. through policy measures) in order to obtain more desirable outcomes. For more accurate estimates of the Sobol indices, M must be increased substantially, thereby greatly increasing also the computational cost. We refrain from doing so in this study and give more comments on the associated computational cost in the Discussion.

The uptake by the population of the interventions plays a crucial role whenever this parameter is part of the strategy. In these strategies (i.e. FC, IL, PO), the uptake parameter is responsible for at least 30% of the QoI variance and, in case of Intermittent Lockdown it is responsible for about 70% of the variance. The intervention_effect, lock_effect and pl_intervention_effect_hi parameters are also important, although they are not always the main driving factor. For Flattening the Curve its Sobol index is higher than that of uptake, whereas for Intermittent Lockdown uptake has a much higher Sobol index (indicating that it is more important) than lock_effect. The population uptake and the effect of the lockdown seem to be of similar importance for the Phased Opening strategy.

In case of Contact Tracing circa 50% of the QoI variance is determined by the delay between becoming infectious and being identified as such (if at all), i.e. the inverse of the rate per day at which infected individuals are being traced (parameter trace_rate_I). After infected agents have been identified, the reduction of their average contact rate is also important.

The probability of tracing exposed individuals (CT strategy) and the interval between subsequent lifts (PO strategy) give only a small contribution to the model output variability. The intrinsic stochasticity of the model instead does not induce much variability in the model output as the 95%-CI of its Sobol index always includes 0 and does not take values above 5%. Similar conclusions hold for the lengths of the lockdowns and subsequent lift periods in the IL strategy.

The total Sobol indices give qualitatively the same outcomes but highlight higher order interactions involving lock_effect and trace_prob_E (see S3 Fig). The differences between the first order and the total Sobol indices suggest that the most important second-order interactions appear to be those between lock_effect and uptake in the IL strategy, and between trace_prob_E and trace_rate_I in CT. Assessing these interaction effects in detail is computationally very expensive, therefore we do not carry out this assessment here. However, the scatter plots for the safe operating space give some idea of the interaction effects (see next section).

The Sobol indices for the second QoI show qualitatively similar results, see S4 and S5 Figs.

Given the knowledge on the main driving factors of each strategy, we want to know which combinations of values for these input parameters result in, for instance, a number of prevalent cases in IC (first QoI) below the IC capacity. This information can be used to devise policy measures that would effectively move the corresponding input distributions towards an area in the parameter space where the strategy is robust, i.e. towards the safe operating space. The information can be obtained by means of a scatter plot of the QoIs as functions of the values of the two or three input parameters with the highest Sobol indices.

In case of Flattening the Curve we visualize the two selected QoIs as functions of the most important parameters according to the Sobol indices, i.e. intervention_effect and uptake, see Fig 4. The first QoI stays below the IC capacity for high population uptake and strong intervention (recall that stronger intervention is associated with lower values of intervention_effect). If uptake is less than circa 85% or if intervention_effect is higher than 35%, an exceedance of the IC capacity is to be expected in the model.

The two main drivers of the strategy Contact Tracing are trace_rate_I and trace_contact_reduction, while trace_prob_E has higher relevance when considered in combination with the other parameters. Therefore, we ignore the seed parameter as the variance it induces is small (the 95%-CI of its Sobol index being very tight around 0) and split our data into quartiles (very low, low, high and very high) of trace_prob_E. The scatter plots of the resulting sub-sets of data of the first QoI are provided in Fig 5. The same analysis for the second QoI reveals the same qualitative results and is therefore not reported here.

At low values of trace_rate_I (i.e. in case of a long delay in the identification process), the first QoI is predicted to be above the threshold independently from trace_contact_reduction. This is slightly mitigated in the plots corresponding to the high and very high quartiles of trace_prob_E, revealing the important interaction between these two parameters captured by the total Sobol index. It is therefore paramount to have an efficient tracing system for this strategy to be effective.

Out of the five considered parameters of the strategy Intermittent Lockdown, only two are important according to the estimated Sobol indices. These are uptake and lock_effect. The scatter plots of the QoIs as functions of these two parameters (see Fig 6) confirm that high levels of uptake are necessary when adopting this strategy. They also show that, as the level of transmission allowed by the lockdown increases (i.e. higher values of lock_effect), high levels of uptake are not enough to keep the first QoI below its threshold. Very little variability is induced by the other parameters.

Similar to Contact Tracing, the Phased Opening strategy has two parameters responsible for most of the output variance: uptake and pl_intervention_effect_hi. The interval between consecutive phases, i.e. intervention_lift_interval, has a small Sobol index, whereas the Sobol index of the random seed is very close to zero. We therefore apply a similar approach: we ignore the random seed and divide the data into quartiles of intervention_lift_interval. We show the scatter plots of the resulting sub-sets of the first QoI in Fig 7. The same analysis for the other QoI reveals the same qualitative insights and is therefore not reported here.

There is a clear gradient visible in Fig 7 with lowest QoI values obtained for high values of uptake and for low values of pl_intervention_effect_hi (corresponding to more restrictive lockdowns). Higher QoI values are obtained when either the population uptake or the rigor of the lockdown decreases. These two factors alone though are not enough to ensure that the IC capacity is not exceeded. In fact all of our simulations in the quartile corresponding to very low values of intervention_lift_interval go beyond the IC capacity (top left panel in Fig 7). By increasing intervention_lift_interval, more simulations stay below the IC capacity threshold.