^{1}

^{2}

^{1}

^{1}

^{2}

^{1}

^{3}

The authors have declared that no competing interests exist.

Many countries are currently dealing with the COVID-19 epidemic and are searching for an exit strategy such that life in society can return to normal. To support this search, computational models are used to predict the spread of the virus and to assess the efficacy of policy measures before actual implementation. The model output has to be interpreted carefully though, as computational models are subject to uncertainties. These can stem from, e.g., limited knowledge about input parameters values or from the intrinsic stochastic nature of some computational models. They lead to uncertainties in the model predictions, raising the question what distribution of values the model produces for key indicators of the severity of the epidemic. Here we show how to tackle this question using techniques for uncertainty quantification and sensitivity analysis. We assess the uncertainties and sensitivities of four exit strategies implemented in an agent-based transmission model with geographical stratification. The exit strategies are termed Flattening the Curve, Contact Tracing, Intermittent Lockdown and Phased Opening. We consider two key indicators of the ability of exit strategies to avoid catastrophic health care overload: the maximum number of prevalent cases in intensive care (IC), and the total number of IC patient-days in excess of IC bed capacity. Our results show that uncertainties not directly related to the exit strategies are secondary, although they should still be considered in comprehensive analysis intended to inform policy makers. The sensitivity analysis discloses the crucial role of the intervention uptake by the population and of the capability to trace infected individuals. Finally, we explore the existence of a safe operating space. For Intermittent Lockdown we find only a small region in the model parameter space where the key indicators of the model stay within safe bounds, whereas this region is larger for the other exit strategies.

Many countries are currently dealing with the COVID-19 epidemic and are looking for an exit strategy such that life in society can return to normal. For that purpose computational models are used to predict the spread of the virus and to assess the efficacy of policy measures before putting them into practice. These models are subject to uncertainties (due to, for instance, limited knowledge of the parameter values), which can lead to a large variability in model predictions. It is therefore fundamental to assess which range of values a model produces for key indicators of the severity of the epidemic. We present here the results of the uncertainty and sensitivity analysis of four exit strategies simulated with an individual-based model of the COVID-19 transmission. As key indicators of the severity of the pandemic we consider the maximum number of cases in intensive care and the total number of intensive care patient-days in excess. Our results show the crucial role of the intervention uptake by the population, of the reduction in the level of transmission by intervention and of the capability to trace infected individuals.

This is a

Many countries are currently dealing with the COVID-19 epidemic and are searching for an exit strategy such that life in society can return to normal. However, in absence of an effective curative treatment and, until recently, of an effective vaccine, non-pharmaceutical interventions have been used to keep case numbers as low as possible. In the past there have been numerous other epidemics during which government actions were required to protect the population. Examples are the influenza pandemic (also known as Spanish flu) in 1918 or the more recent Mexican flu (or swine flu) in 2009 [

Since computational modeling of the epidemic has come to play a significant role for informing policy, it is important to assess the uncertainties of the models and of their predictions. Such uncertainties can stem, for instance, from limited knowledge about the values of the input parameters or from the intrinsic stochastic nature of part of the computational models. The presence of any of these uncertainties leads to uncertainties in the model predictions. A central question is therefore what distribution of values is produced by the model for key indicators of the severity of the epidemic and of the intensity of interventions.

In this study we present results from an analysis of uncertainties and sensitivities of an agent-based model of the COVID-19 epidemic [

The UQ and SA frameworks employ a probability distribution—as opposed to a single value—to describe each input parameter. These distributions are typically determined from available data or from expert knowledge. With proper probability distributions assigned to each input parameter, the UQ and SA frameworks can be used to assess whether exit strategies are robust. A useful criterion for robustness is that, given the model input distributions, the 95^{th} percentile of a chosen output quantity—usually termed Quantity of Interest (QoI)—remains below a critical threshold. Examples of QoI in the context of epidemic modeling are the peak number of COVID-19 patients in intensive care (IC) units and the total number of fatalities due to COVID-19. For other QoIs, e.g. life years gained, a more relevant criterion is that the 5^{th} percentile stays above a minimum value. This can be dealt with in an analogous manner.

The aim of this work is to perform a model-based quantitative analysis of uncertainties and sensitivities for the computational representations of four exit strategies, and assess the uncertainties in model simulation results. We show how such analysis can be performed by means of computational methods and concepts from the fields of uncertainty quantification and sensitivity analysis, and what kind of insights can be obtained. UQ is important when decision makers have to choose between interventions, guided by model-based predictions of what the effects of these interventions will be. Consider a situation where, looking at a single prediction based on the best point estimate for the model input parameters, intervention A appears more attractive than intervention B (e.g. because the predicted peak number of COVID-19 patients in IC is lower under A). However, when taking parameter uncertainties into account, UQ may show that intervention A also gives a higher probability of disastrous outcomes than B (e.g. higher chance of peak numbers that overwhelm IC capacity). This latter insight can be reason to choose B over A. SA provides a different kind of insight: it shows which parameters have the most effect on the outcome and on its uncertainty, and thereby which parameters are the most important ones to control or influence through policy. For example, in case of the Contact Tracing strategy considered in this paper, is it more effective to spend additional resources on tracing

In order to perform our analysis we consider the spread of the COVID-19 disease in the Netherlands in the context of an open-source agent-based model with geographical stratification [

In what follows, we provide an overview of the computational model used in our analysis and the exit strategies implemented in the model. Next, we summarize some key concepts of uncertainty quantification and describe different sources of uncertainties. We discuss also the chosen SA method and the quantities of interest selected for our study. We conclude this section with a description of the computational UQ and SA framework that we employ.

We employ the publicly available virsim model [

Values represent the mean and 95%-confidence interval (95%-CI) of the input distributions.

Input parameter | Strategy | Distribution | Mean | 95%-CI |
---|---|---|---|---|

FC | Beta( |
0.35 | [0.27, 0.44] | |

FC, PO, IL | Beta( |
0.89 | [0.71, 0.99] | |

CT | Γ(shape = 2, scale = 0.2) | 0.40 | [0.05, 1.11] | |

CT | Beta( |
0.25 | [0.04, 0.58] | |

CT | Beta( |
0.83 | [0.59, 0.98] | |

IL | Γ(shape = 20, scale = 2) | 40.0 | [24.4, 59.3] | |

IL | Γ(shape = 15, scale = 1) | 15.0 | [8.4, 23.5] | |

IL | Beta( |
0.25 | [0.15, 0.37] | |

PO | Γ(shape = 25, scale = 2) | 50.0 | [32.4, 71.4] | |

PO | Beta( |
0.25 | [0.15, 0.37] | |

non-policy-related | Γ(shape = 25, scale = 0.2) | 5.00 | [3.24, 7.14] | |

_{0} |
non-policy-related | Γ(shape = 100, scale = 0.025) | 2.50 | [2.03, 3.01] |

^{-1} |
non-policy-related | Γ(shape = 2, scale = 0.05) | 0.10 | [0.01, 0.28] |

non-policy-related | Γ(shape = 17.5, scale = 1) | 17.5 | [10.3, 26.6] |

We consider four exit strategies: Flattening the Curve (FC), Contact Tracing (CT), Intermittent Lockdown (IL) and Phased Opening (PO). Each strategy is part of the model implementation (see

Since this study is of conceptual nature and does not aim to model real-world scenarios as accurately as possible, we simulate a population of 1 million individuals and we focus on the first year after the implementation of a strategy (the idea being that governments will regularly evaluate and adapt the strategy in use). In the

Uncertainties in model output can arise from different sources; we discuss here four main types. The first is parameter uncertainty, referring to uncertainties in model parameters whose values can be set directly by the model user via the inputs of the computational model. An example is the reduction of the transmission rate due to the introduction of an intervention. For the COVID-19 epidemic, a significant decrease in the transmission rate following the adoption of such measures has been recorded [

The second type of uncertainty, which we call intrinsic uncertainty, arises when a computational model is inherently stochastic. In epidemiology, many models are agent-based and possess internal stochasticity, for instance in the randomized interactions between agents. Model users often have little control over such internal stochasticity as they can typically only set the seed of the random number generator at the start of a simulation [

The third type is model-form uncertainty, referring to uncertainty or errors in the structure of the model itself (e.g. due to transmission mechanisms not represented in the model). This type of uncertainty cannot be analyzed by changes in the model inputs but requires a comparison either with independent observation data or with other models—as done for example in climate science [

Lastly, initial condition uncertainty is due to the inaccuracies in the specification of the initial state of the model (i.e., the state of the simulated population at the start of the model run). Since we consider here model outputs which are independent from the specific timing of e.g. epidemic peaks, this type of uncertainty is not important in our analysis.

In this study we analyze the parameter and intrinsic uncertainties by means of non-intrusive UQ methods, which means that we treat the computational model (i.e., the virsim model) as a black box. We extend the notation from the introduction to

In order to assess which parameters create the most uncertainty in the model output _{i}) is defined as _{i} and _{1}, _{2}, … except _{i} (and similarly for Var_{i} and Var_{∼i}). If _{i} is close to 1, it means that the variance of _{i}. The overall effect on the model output of all parameter combinations involving _{i} is given by the total Sobol index

Regarding the assumed mutual independence of the input parameters, there might be dependencies when parameters are actually estimated from data. However such dependencies are not implemented in the computational model, hence the selected input parameters (see below for more details) are effectively treated as mutually independent. We refer to

In our analysis we consider the model predictions for the number of incident and prevalent individuals in the population that require IC admission. As IC capacity is limited, the question whether the IC capacity will be exceeded (and if so, by how much) according to model predictions is clearly important. To investigate this, for each simulation we consider two quantities of interest (QoI):

the maximum of the moving average of the prevalent cases in intensive care (averaging window = 30 days)

the total number of IC patient-days in excess of IC bed capacity (referred to as “IC patient-days in excess” from hereon).

The first QoI shows the peak value of the number of COVID-19 patients in IC units, giving an indication of the intensity of an outbreak. We apply a moving average to focus on longer-term trends, filtering out short-term “noisy” variations. When analyzing robustness, a natural threshold for this QoI is the available IC capacity, which may vary from country to country and from month to month. In our analysis we assume the IC capacity to be constant in time and we consider the Netherlands as reference country. De Vlas and Coffeng [

The second QoI quantifies by how much the total IC capacity is overburdened. It is defined as

We note that the two QoIs as defined here do not depend on time. The first QoI is defined as a maximum over the time interval of simulation, whereas the second QoI is defined as a summation over the same time interval. Thus, a single model simulation yields a single, time-independent value for each QoI.

The implementation of a strategy in the computational model is determined by a set of input parameters for the model. Some of the parameters can be controlled (up to a certain degree) by policy makers. These are parameters related to social aspects of the population—e.g. social distancing—and to the availability of resources, e.g. to test and track infected individuals. They can be controlled to some extent by government authorities through imposing stricter (or less strict) rules for e.g. social distancing, or through making more (or less) resources available. We call them policy-related parameters and we refer to the uncertainties generated by these parameters as policy-related uncertainties.

Below we provide details about which policy-related parameters are treated as uncertain, together with the rationale behind the probability distributions chosen for them. The distributions, together with their mean and 95%-confidence intervals are provided in

We use Beta distributions for those parameters that are naturally restricted to values on a finite interval, such as those representing probabilities or percentages. We remark that this allows us to use distributions that are very different from the uniform distribution. With the hyperparameters used here, the probability density of the Beta distribution decreases to zero towards the boundaries of the support. By contrast, the uniform distribution attributes equal probability to any value within the support of the distribution. For parameters defined as a time scale or a rate we choose the Γ distribution as it has one unique peak (with the hyperparameters used here) and a semi-infinite domain (all non-negative values).

Our choice of the distributions was guided by expert knowledge. Ideally, as more data become available the initial guesses for those parameters that can be directly estimated could be updated to better reflect the data (although this has not been done here). However, this is not always possible as there may be parameters that cannot be estimated directly from data. Furthermore, parameters can be model specific. Relatedly, some of our selected distributions are not transferable to other transmission models if these have explicit representations (with dedicated parameters) of processes only implicitly represented in virsim. For instance, in the virsim model the parameter for the effect of the intervention in Flattening the Curve implicitly includes the effect of different measures (e.g. social distancing, working from home and closure of schools), therefore it cannot be directly estimated from data for every plausible combination of interventions. Hence it is advisable to combine expert knowledge and calibration on the available data to determine the most appropriate input distributions.

the effect of the intervention (model parameter

the uptake of the intervention by the population, model parameter

the delay between becoming infectious and being identified as such (if at all); represented by the inverse of the model parameter

the probability of an infected contact to be identified before the person turns infectious; represented by the model parameter

the quality of the isolation and its effect on transmission; represented by the model parameter

the duration of the lockdown and of the following lift of measures represented by the model parameters

the effect of the lockdown (model parameter

the

the time interval between one phase and the next, determined by

the effect of the measures in the areas where the lift has not been applied yet, represented by the model parameter

the uptake of the intervention by the population, model parameter

Besides the parameters that can be influenced by policy, discussed in the previous section, the computational model has other parameters that we include in our analysis. Examples are the reproduction number of the virus and the duration of infectiousness. Because these parameters have to be estimated from data, their values can be rather uncertain, especially when available data is scarce (for instance in the early stages of an epidemic with a new virus). For some parameters, care has to be taken when sampling them from probability distributions, since certain basic characteristics, like the doubling time, have to match the time evolution of the epidemic as captured in the (real-world) observation data. We chose distributions whose mean is (almost) the value obtained in a previous quantification of the model [

Altogether, given the setup of the virsim model, we consider the following non-policy-related uncertain parameters:

the average duration of infectiousness (model parameter

the reproduction number _{0}_{0}_{0} and the average duration of infectiousness;

in order to consider uncertainties in how the incubation period varies among the population, we sample the shape parameter of the distribution of

The virsim model is stochastic, and as discussed in an earlier section, this internal stochasticity is a source of uncertainty of the model output. It is not possible to choose the probability distribution of the internal (or latent/hidden) random variables of the model, or to set them by hand to specific values according to some sampling plan. The only form of control as model user is to pick the seed for the pseudo-random number generator at the start of the simulation. An example of uncertainties in the model output due to the internal stochasticity is provided in

Furthermore, when using models with a geographical structure like the virsim model, uncertainties can arise from the level of geographical mixing that is allowed in the model. For sake of simplicity we do not consider such uncertainties in the present study, but they should be considered in more comprehensive studies if these are intended to inform policy makers.

We report in

If the dependence of the QoIs on the parameters is smooth and the number of uncertain parameters is not too high, the propagation of uncertainties from parameters to QoIs can be assessed with techniques such as Polynomial Chaos Expansion and Stochastic Collocation [

For the SA, we compute the first order Sobol indices with the cost effective algorithm of Saltelli [^{2}

Sampling and post-processing analysis are done using the Monte Carlo sampler of the publicly available Python library EasyVVUQ [

To provide intuition for the behaviour of the model and the QoIs defined earlier, we show in

Left: time series of the prevalent cases in intensive care. Middle: the resulting moving average of the number of IC prevalent cases. Right: the corresponding time series of the total number of IC patient-days in excess. All values on the vertical axis are per million capita. Black circles in the middle and right panel indicate the QoIs for the different simulations.

As first step of our UQ analysis, we construct the empirical cumulative distribution functions (cdfs) of our QoIs. For any threshold value

Top row: results with policy-related and non-policy-related uncertainties. Bottom row: outcomes with uncertainties only in the policy-related parameters and in the random seed. The vertical dotted black line indicates the maximum IC capacity, while the thinner colored lines denote the 95%-CI given by the Dvoretzky-Kiefer-Wolfowitz inequality. Note that the cdfs for the second QoI (right column) do not start from zero probability because the distributions have a non-zero probability that the number of IC patient-days in excess is zero.

We observe the important fact that, with the given distributions of the input parameters, none of the analyzed strategies is robust. The probability that the number of prevalent patients in intensive care is larger than the IC capacity is rather high and only Contact Tracing gets close to a probability of 50%. This shows that the assumed input distributions for Flattening the Curve, Intermittent Lockdown and Phased Opening correspond to interventions that are not sufficiently restrictive to stay below the threshold, as far as the model is concerned.

We note that the cdfs of the first QoI for CT and IL increase more gradually compared to FC and PO (implying that the variance of the first QoI is larger for CT and IL than it is for FC and PO). Furthermore, it can be seen that the shape of the cdfs is only weakly affected by non-policy-related parameters. Therefore, when searching for the parameters responsible for most of the output variability, i.e. for the sensitivity analysis, these parameters might be kept fixed to reduce the computational burden. They should be included however when determining the minimum level of e.g. intervention or uptake, required for the QoIs to stay below their threshold with 95% probability.

In

The length of the bars indicate the mean values, while the thinner lines display the 95% confidence interval. We color in orange the

From the width of the intervals, it can be seen that the number of MC samples (

The uptake by the population of the interventions plays a crucial role whenever this parameter is part of the strategy. In these strategies (i.e. FC, IL, PO), the

In case of Contact Tracing circa 50% of the QoI variance is determined by the delay between becoming infectious and being identified as such (if at all), i.e. the inverse of the rate per day at which infected individuals are being traced (parameter

The probability of tracing exposed individuals (CT strategy) and the interval between subsequent lifts (PO strategy) give only a small contribution to the model output variability. The intrinsic stochasticity of the model instead does not induce much variability in the model output as the 95%-CI of its Sobol index always includes 0 and does not take values above 5%. Similar conclusions hold for the lengths of the lockdowns and subsequent lift periods in the IL strategy.

The total Sobol indices give qualitatively the same outcomes but highlight higher order interactions involving

The Sobol indices for the second QoI show qualitatively similar results, see

Given the knowledge on the main driving factors of each strategy, we want to know which combinations of values for these input parameters result in, for instance, a number of prevalent cases in IC (first QoI) below the IC capacity. This information can be used to devise policy measures that would effectively move the corresponding input distributions towards an area in the parameter space where the strategy is robust, i.e. towards the safe operating space. The information can be obtained by means of a scatter plot of the QoIs as functions of the values of the two or three input parameters with the highest Sobol indices.

In case of Flattening the Curve we visualize the two selected QoIs as functions of the most important parameters according to the Sobol indices, i.e.

The black dots show the simulations whose QoI value is below or equal to the IC capacity or there are no IC patient-days in excess.

The two main drivers of the strategy Contact Tracing are

The black dots show the simulations whose maximum value is below or equal to the IC capacity. The plots correspond to quartiles of

At low values of

Out of the five considered parameters of the strategy Intermittent Lockdown, only two are important according to the estimated Sobol indices. These are

The black dots show the simulations whose maximum value is below or equal to the IC capacity or there are no IC patient-days in excess.

Similar to Contact Tracing, the Phased Opening strategy has two parameters responsible for most of the output variance:

The black dots show the simulations whose maximum value is below or equal to the IC capacity. The plots correspond to quartiles (top left: very low; top right: low; bottom left: high; bottom right: very high) of

There is a clear gradient visible in

The aim of this work was to perform a model-based quantitative analysis of the uncertainties and sensitivities of a number of selected exit strategies for the COVID-19 epidemic, thereby showing how such an analysis can be carried out and what decision-relevant insights can be obtained from it. We discussed how the analysis can be approached using methods and concepts from the field of Uncertainty Quantification and Sensitivity Analysis. UQ and SA make it possible to give probabilistic answers to policy questions, such as the probability that the number of prevalent cases in intensive care will not exceed the IC capacity. This helps policy makers to gauge the uncertainties in the model predictions and to take more informed decisions.

As concrete examples of decision-relevant insights that UQ and SA can deliver, we mention two observations taken from our numerical results.

We demonstrated computational techniques that can be employed to assess the uncertainties and identify the sensitivities of each strategy. In particular, we examined the empirical cumulative distribution function of two quantities of interest obtained from the model output: the maximum number of prevalent cases in IC and the total amount of IC patient-days in excess of IC bed capacity. We also identified the input parameters responsible for most of the output uncertainty (namely

Given the probability distributions that we chose for the uncertain model parameters, the Contact Tracing strategy appears to have the most potential to avoid exceeding IC capacity (see

None of the strategies analyzed here satisfied our criterion of robustness with the given input distributions. This means that the probability that the number of prevalent patients in IC is larger than the IC capacity is rather high for all four exit strategies. To achieve more satisfactory performance (as far as the model and the chosen QoIs are concerned), parameter distributions corresponding to stricter interventions are needed. This would require that the effects of policy measures can “push” the bulk of the parameter distributions to more favorable values compared to the distributions used in our analysis here. In particular the insights about the safe operating space can be useful to determine how the parameters distributions might need to be modified in order to obtain more desirable outcomes, e.g. towards stricter interventions in case of Flattening the Curve or towards longer intervals between consecutive phases for Phased Opening.

The analysis presented here can be extended to a broader set of models and diseases. Since we used non-intrusive methods, the same type of analysis can be applied to a different transmission model for COVID-19 or to a computational model for a different epidemic. Furthermore, the set of methods used for analysis can be enlarged in several ways. Many UQ and SA techniques are available tackling different aspects of the problem according to the type of information that one would like to obtain; see, e.g., [

A similar model-based analysis of uncertainties has been performed by Davies et al. [

It is important to realise that UQ and SA results are conditioned on the choices of parameter distributions and should therefore be interpreted with caution. As such, our results should not be interpreted as a definitive formal ranking of the analyzed exit strategies, as these strategies might show better or worse performance when considering different distributions. We aimed at picking plausible distributions, however we do not claim a homogeneous level of rigor among the choices that were made. On a related note, we assumed in this study that once the strategy is started it is not changed, implying that input parameters (or their distributions) do not change over time. Furthermore, we limited the scope of our analysis to a small number of key parameters for which we assumed specific probability distributions. However, the computational model has additional parameters (relating both to policies and to other aspects), which were kept fixed here but could be added to the set of uncertain parameters in a more comprehensive UQ analysis.

A concrete example of these additional parameters is formed by the parameters specifying the geographical stratification. On one hand the geographical stratification makes the model more realistic and allows for heterogeneity in the population and the evaluation of regional measures; on the other hand it adds parameters to the model, making it more uncertain. Thus, the importance of the geographical stratification for policy making should be assessed case by case. In our case it was important because one of the considered strategies (Phased Opening) includes regional measures. If regional interventions are not under consideration, the benefits of the geographical stratification may not outweigh the additional uncertainty implied by the (uncertain) stratification parameters. This relates to the more general issue of model selection and of balancing model complexity against model uncertainty and risk of overfitting, a well-known challenge in mathematical and statistical modeling.

For the policy-related parameters, the chosen distributions correspond to the (assumed) effects of policy measures in the real world. The feasibility of implementing such measures is a different matter, beyond the realm of mathematical and computational modeling and therefore not considered here, but important nonetheless. As a concrete example,

We conclude with some remarks about scaling up the analysis performed here to more extensive assessments and to more complex models (with higher computational cost). Scaling up will enable fast, frequent and comprehensive analysis of uncertainties and sensitivities in epidemiological models. Executing such analysis in a timely fashion is essential for it to be useful for policy makers. In this study we limited the analysis to a handful of uncertain parameters (less than 10), keeping all other parameters fixed. A more comprehensive study, on a larger set of parameters, will be computationally more demanding as it typically requires more model runs. While the execution of a single simulation of the virsim model takes 1–2 minutes on a laptop, more complex models have higher computational costs. Furthermore, (tens of) thousands simulations are required for thorough analysis of model uncertainties and sensitivities. Thus, access to sufficient computational resources is important to scale up.

For the analysis reported here, we had access to a supercomputer of the Poznan Supercomputing and Networking Center. A single campaign with circa 10000 runs for the SA of an exit strategy took several hours in total (including the time needed for the job submission to the supercomputer, parallel execution of the model runs on a single node with 28 cores, and retrieval of the results). If quantification of uncertainties is to be performed frequently and rapidly (e.g. in an “operational” setting with a daily or weekly cycle of producing forecasts with quantified uncertainties, or for weekly re-evaluation of a multitude of policy options), a dedicated computational infrastructure is recommendable to have uninterrupted access and to avoid long queuing times for compute jobs.

Besides access to computational resources, software suitable for efficient UQ and SA is needed. The open source VECMA toolkit used in this study is developed for use on high-performance computing platforms. Last but not least, a dedicated team with combined expertise (UQ and SA; epidemiology and computational modeling; high-performance computing and software) will be central to successful upscaling and thereby to support policy making with timely information about uncertainties and sensitivities of model results.

In this study we analyzed the uncertainties and sensitivities of an agent-based transmission model for the COVID-19 epidemic under four different exit strategies. Our analysis showed that the uncertainties in the model simulation results for each considered exit strategy are substantial. They were found to be mostly generated by uncertainties in the parameters directly related to the strategy itself (such as implementation and uptake of the strategy) rather than uncertainties due to other factors (such as duration of infectiousness). With the parameter distributions that we choose, the Contact Tracing strategy was the most effective. Finally, because we used non-intrusive methods, our analysis can easily be extended to other strategies as well as to other computational models and epidemics.

Overview of the values of the parameters in the SEIR model. In the last column we indicate the respective parameter in the computational model (or which computational parameter is affected).

(PDF)

(PDF)

(PDF)

(PDF)

Distributions for the uncertain input parameters in case of Flattening the Curve (top left), Contact Tracing (top right), Intermittent Lockdown and Phased Opening (middle), and for the biology-related parameters (bottom).

(TIF)

Example of strategy outcome variability due to intrinsic stochasticity of the virsim model. The shaded gray area denotes the interval between the 5^{th} and the 95^{th} percentiles out of 100 realizations of the FC strategy with same policy- and non-policy-related parameters but different

(TIF)

Total Sobol indices of the first QoI (the maximum number of patients in IC). The length of the bars indicate the mean values, while the thinner lines display the 95% confidence interval. We color in orange the

(TIF)

First order Sobol indices of the second QoI (the total number of IC patient-days in excess of IC bed capacity). The length of the bars indicate the mean values, while the thinner lines display the 95% confidence interval. We color in orange the

(TIF)

Total Sobol indices of the second QoI (the total number of IC patient-days in excess of IC bed capacity). The length of the bars indicate the mean values, while the thinner lines display the 95% confidence interval. We color in orange the

(TIF)

The calculations were performed at the Poznan Supercomputing and Networking Center.

Dear Dr Gugole,

Thank you very much for submitting your manuscript "Uncertainty quantification and sensitivity analysis of COVID-19 exit strategies in an individual-based transmission model" for consideration at PLOS Computational Biology.

As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. In light of the reviews (below this email), we would like to invite the resubmission of a significantly-revised version that takes into account the reviewers' comments.

The reviewers were in general enthusiastic about the topic and the quaility of writing. I too think that uncertainty analysis in modelling policy advise is important, but I think the manuscript currently does not discuss practical implementation sufficiently. It is said that the model is conceptual, and are an illustration of the UQ and SA approaches, but the results and especially discussion are pretty much focussed on the results of the COVID-19 control scenarios, and less about what the UQ and SA would contribute to the decision making process. I think that should be central: when and why use UQ, and when and why use SA, and how does this improve decisions about the right policy to choose? After all, the manuscript was submitted for the Methods section, and although COVID-19 itself is topical, the strategy comparison in this manuscript is not anymore, so it really should serve as an illustration.

We cannot make any decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript is also likely to be sent to reviewers for further evaluation.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to the review comments and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Please prepare and submit your revised manuscript within 60 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. Please note that revised manuscripts received after the 60-day due date may require evaluation and peer review similar to newly submitted manuscripts.

Thank you again for your submission. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Don Klinkenberg

Associate Editor

PLOS Computational Biology

Virginia Pitzer

Deputy Editor-in-Chief

PLOS Computational Biology

***********************

Reviewer's Responses to Questions

Reviewer #1: This is a very well-written paper and I read it with enthusiasm. I had a few suggestions:

1. The intermittent lockdown is assumed to go into effect periodically with a prespecified duration of lockdown and then a prespecified duration of opening. I am not sure if this is realistic. If possible, I'd suggest modeling this on/off decision as a function of some epidemic threshold; for example, lockdown when the number of COVID-19 patents in ICU passes a threshold T1 and reopen when the number of COVID-19 patients in ICU goes below a threshold T2. This is more consistent with the way decisions about physical distancing measures have been made and modelled in existing studies.

2. Please explain briefly why the 'random number seed' is also considered as parameter (line 346). I am assuming this is to ensure that the Sobel index of the random number seed is negligible since otherwise, it suggests a problem with the random number generator.

3. There are other approaches for sensitivity analysis and quantification of uncertainty such as the use of partial-rank correlation:

Reviewer #2: This manuscript provides a well-done and much needed discussion of

public-health measures against the COVID-19 pandemic. It is extremely

timely and provides well-founded answers to the highly relevant

question how effective the various policy measures are. The numerical

results can be improved given more computational resources.

Major points that should be addressed in a revision are the following.

Regarding the computational model (page 3), it would be convenient for

future readers to show the (well-known) SEIR model equations and

especially how the geographical stratification works. This could also

be done in an appendix, but I believe it would be preferable to show

the basic equations and the geographical stratification at the

beginning to help the reader to understand how and where the various

parameters enter the system of equations. In this regard, it would

also be useful to move Table 1 up close to the equations so that the

reader can find this important information, i.e., the system of

equations and all parameters, in one place at the beginning.

On page 3, it is said that "this study is conceptual in nature".

Still, it does a very good job at trying to be realistic. A

discussion of what is still needed to move from a concept to a

realistic treatment would be beneficial in order to help the reader

understand any shortcomings if they exist.

On pages 6 ff., it is mentioned that Beta and Gamma distributions are

assumed for the parameters and some justification is given on page 6.

However, would it be possible to provide more specific justifications

on these pages where the parameters are discussed?

In Table 1, references (if available) to justify the choice of the

parameter values would be very welcome and add a lot to the

discussion.

On page 9, it is mentioned that the geographical stratification in the

model adds to the uncertainty in the model via additional parameters.

More unknown parameters mean more uncertainties. Therefore the

question arises if the additional geographical parameters are indeed

advantageous in the sense that they yield an improved model. Does the

improvement in the model justify the complications? These questions

should be discussed.

On page 12 it is mentioned that the number M of runs would need to be

increased substantially for more accurate estimates of the Sobol

indices. Therefore the computational cost (CPU hours) should be

discussed a bit more detailed. The discussion in the discussion

section comes a bit late and I think it would be good to move some

statements about the computational cost up.

On page 17, the references [7, 8] regarding the use of Bayesian

inference to estimate the distributions of the input parameters are

very general. Also reference [35] is cited in the context that UQ

techniques have (surprisingly) seldom been applied to epidemiological

models. An exception I could find is the following work, where such

Bayesian calculations were performed already a year ago; the authors

might want to consider citing it, also in the context of Table 1.

[Leila Taghizadeh, Ahmad Karimi, and Clemens Heitzinger. Uncertainty

quantification in epidemiological models for the COVID-19

pandemic. Computers in Biology and Medicine, 125(104011):1--11, 2020.]

In the discussion on page 18, the computational cost is approximately

given. More data would be very useful. How long does one simulation

take? Also, is the geographic stratification in the simulations worth

it? It has the drawback that it increases computational cost and adds

more (uncertain) parameters. Also it seems as if the geographical

part of the model was left constant in this work anyway.

Reviewer #3: Gugole et al. present a thorough uncertainty quantification and sensitivity analysis of an agent-based model of SARS-CoV-2 transmission. The model is based on the Netherlands, but the paper is really focused on describing these methods carefully and discussing the interpretation of their output, rather than attempting to make realistic projections. They consider four strategies for reopening, and find that, for each strategy, 2-3 key parameters largely determine whether ICU capacity will remain manageable, and that these key parameters are always policy-related parameters.

I very much enjoyed reading the paper. It is well written, explains the methods clearly, and would be a very useful read for an infectious disease modeler (or any modeler) who wishes to learn and employ UQ and SA in their work. I only have a few minor comments, detailed below:

Methods and model

• Line 73: you could refer to S1 Fig here, as the reader may wish to visualize the distributions of these parameters.

• What sampling method did you use to sample from the parameters distributions in the UQ?

• Line 208: write the date to which you reconstruct rather than ‘present time’

• Please give references for the distributions of the non-policy related parameter ranges you use, e.g. R0, infectiousness duration etc. If possible, it would be nice to see references supporting your choice of distributions for the policy related parameters, too. I think at the moment, the rather hand-wavy justification for these distributions may be the weakest part of the manuscript, although I appreciate that most of these parameters may not have much literature to support your choices.

• Line 337: Could you explain your choice of 1000 simulations for the UQ and the choice of M=2000 for the SA.

• On this subject, it’s not completely clear whether you do 1000 simulations per parameter set or 1000 overall? I’m assuming the former, but probably best to make this clear around line 338

• In Table 1, could you indicate in some way which type of parameter each is (i.e. policy-related or other), and if policy-related, which strategies it pertains to?

Results

• Could you show a supplementary figure of one reopening scenario, similar to Figure 1, but where each line uses the same parameter set? Perhaps with more than four lines – something like a spaghetti plot would work well. This would help the reader to get a sense of how stochastic the model is.

• Line 407: could you include the SA for the other QoI as a supplementary figure?

• Figure 3: Could you show the higher order terms on this plot? Judging from the x-axis they must be pretty small, but it would still be nice to see it.

• Line 443: Could you show the second order interaction terms in a supplementary table or figure (e.g. a heat-map)?

Discussion

• Line 512: I’m not sure most effective is the correct term here, because as you say, it depends on the parameters. Perhaps it would be better to say has the most potential to avoid exceeding ICU capacity, or something like that?

**********

The

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

PLOS authors have the option to publish the peer review history of their article (

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool,

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at

Submitted filename:

Dear Dr Gugole,

We are pleased to inform you that your manuscript 'Uncertainty quantification and sensitivity analysis of COVID-19 exit strategies in an individual-based transmission model' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology.

Best regards,

Don Klinkenberg

Associate Editor

PLOS Computational Biology

Virginia Pitzer

Deputy Editor-in-Chief

PLOS Computational Biology

***********************************************************

Reviewer's Responses to Questions

Reviewer #1: The authors have addressed all my comments and concerns in this revision.

Reviewer #2: The points raised in my review were addressed.

**********

The

Reviewer #1: Yes

Reviewer #2: Yes

**********

PLOS authors have the option to publish the peer review history of their article (

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Reviewer #1: No

Reviewer #2: No

PCOMPBIOL-D-21-00514R1

Uncertainty quantification and sensitivity analysis of COVID-19 exit strategies in an individual-based transmission model

Dear Dr Gugole,

I am pleased to inform you that your manuscript has been formally accepted for publication in PLOS Computational Biology. Your manuscript is now with our production department and you will be notified of the publication date in due course.

The corresponding author will soon be receiving a typeset proof for review, to ensure errors have not been introduced during production. Please review the PDF proof of your manuscript carefully, as this is the last chance to correct any errors. Please note that major changes, or those which affect the scientific understanding of the work, will likely cause delays to the publication date of your manuscript.

Soon after your final files are uploaded, unless you have opted out, the early version of your manuscript will be published online. The date of the early version will be your article's publication date. The final article will be published to the same URL, and all versions of the paper will be accessible to readers.

Thank you again for supporting PLOS Computational Biology and open-access publishing. We are looking forward to publishing your work!

With kind regards,

Andrea Szabo

PLOS Computational Biology | Carlyle House, Carlyle Road, Cambridge CB4 3DN | United Kingdom