Prior to any simulator being used as a tool to complement wet-lab investigations, it is critical that the effect of inherent simulation stochasticity on results be understood [5]. To illustrate, in agent-based simulations the use of pseudo-random number generators in dictating agent behaviour can produce different simulation results despite use of identical parameter values. To mitigate the effects of this uncertainty and achieve representative in silico results, replicate simulation runs are necessary. A technique developed by Read et al. [6] is provided that establishes the number of replicate runs required to achieve a desired level of experimental accuracy. In contrast to the other techniques in spartan, this need only be applied to stochastic simulation systems.

Consistency analysis operates by contrasting distributions of simulation responses, all generated using the same fixed set of parameter values and containing identical numbers of simulation samples. By varying the number of samples comprising the distributions, the analysis determines the number required to obtain statistically consistent distributions. Larger sample sizes produce increasingly identical distributions, thereby mitigating the effect of simulation stochasticity on results. In the description by Read et al [6], 20 distributions are generated and contrasted for each sample size, but this can be varied within spartan to suit particular applications.

As an example, one could consider analysing sample sizes of 5, 50, 100, and 300 to determine the number of simulation runs required to mitigate aleatory uncertainty. A set of parameter values is fixed and used for all runs. The researcher performing the analysis must then gather 20 sets of simulation results for each sample size being analysed, each containing that number of results. In this example, where a sample size of 5 is being examined, 20 sets of simulation results should be generated, with each containing 5 sets of simulation results. Where a sample size of 300 is being analysed, each of the 20 sets should contain the results of 300 runs. When this is complete, each sample size is analysed in turn. A distribution of median responses for each simulation run is generated for each of the 20 subsets.

Distributions 2–20 are contrasted with the distribution from the 1st set using the Vargha-Delaney A-Test [14], a non-parametric effect magnitude test that establishes scientific significance by contrasting two populations of samples and returning the probability that a randomly selected sample from one population will be larger than a randomly selected sample from the other. Statistical significance is determined by comparing this result with measures set by the authors of the test: results above 0.71 or below 0.29 indicate a scientifically significant difference between the populations, and 0.5 indicates no difference [14]. However spartan provides the option to change these magnitude effects if required. A suitable sample size is found where there is no statistical difference between the 1st set of results and the remaining 19 sets. This statistical difference can be seen in plots spartan produces for each sample size. A further plot is produced detailing the maximum A-Test score observed for each sample size, easing identification of the number of runs required to minimise difference between simulation results and thus mitigate aleatory uncertainty.

Any biological simulation will feature biologically-derived parameters for which values are fully or partially unknown: some biological values cannot be determined experimentally, whereas others cannot be represented easily within a simulation. For example, diffusion of a chemoattractant could be implemented using a particular mathematical function for which values cannot be verified, as biological quantities cannot currently be measured. Robustness analysis examines the implications of biological uncertainty or parameter estimation on simulation results. Where a simulation is found to be highly sensitive to such parameters caution must be exercised when interpreting results; they may be artefacts of parametrisation rather than representations of the biology [5].

Robustness to parameter perturbation is explored using a ‘one at a time’ approach [6]: each parameter is adjusted independently of others, which retain their calibrated values. The Vargha-Delaney A-Test described previously [14] is employed to determine if changing a parameter value leads to scientifically significant behavioural alteration in contrast to the baseline simulation. This indicates how robust the simulator is to alteration of each parameter, and the points at which parameter perturbations result in significant changes in simulation behaviour. Confidence in the validity of simulation results can be gauged by contrasting this information with biologically accepted ranges of values. The A-Test results for each parameter are presented on a plot, providing easy identification of parameter values that cause a scientifically significant change in simulation response.

Though robustness analysis elucidates affects of perturbing single parameters, it cannot reveal compound effects that occur when two or more are adjusted simultaneously. The effect one parameter has may rely on the value of another. Global sensitivity analyses reveal such effects, showing how different parameters could be coupled, and can indicate the parameters that have the greatest influence on simulation responses. Spartan includes a sampling-based technique that perturbs the values of all parameters of interest simultaneously [6], [7], [13] based on a provided range of values for each parameter. Through simultaneous perturbation of parameters, the results of this analysis are highly representative of simulation dynamics. Highly influential parameters indicate the pathways and components that have a substantial effect on simulation behaviour, and in identifying such relationships, this analysis has the potential to offer unique biological insight into the system the model describes. This has the potential to inform future wet-lab investigations by suggesting specific biological components to focus upon.

A latin-hypercube sampling approach is used to select parameter sets from within these ranges, whilst minimising correlations in parameter values across the sets and ensuring an efficient coverage of parameter space [13]. The methodology used in sampling is described in detail by Read et al [6]. Simulations are executed for each parameter set generated, and simulation response values determined. Where necessary these responses will represent an averaged response over a number of runs for a particular set of parameter values. A plot is produced for each parameter-response pairing, revealing correlations between parameter and response values, which are quantified through calculation of Partial Rank Correlation Coefficients (PRCC) and are reported on the plot header. PRCCs account for non-linear relationships between parameter and response, and correct for the effects of other parameters on the response, giving a robust indication of the effect this parameter has on simulation response although others are also being perturbed.

The extended Fourier Amplitude Sampling Test (eFAST) developed by Saltelli et al [15], [16] also a global analysis technique, is a variance decomposition method that partitions the simulation output variance caused by an alteration in parameter values between the input parameters. This provides a statistical measure revealing the proportion of variance that can be explained by perturbing the value of each factor, and thus determines how sensitive the simulation and biological system is to that parameter. Through applying this technique, highly influential pathways can be identified as potential targets for future laboratory investigation.

For each simulation parameter included in this analysis, a range over which values are to be explored is provided. Taking each in turn, values are chosen for all parameters through the use of sinusoidal functions of a particular frequency through the parameter space, with the frequency of the parameter of interest being varied greatly to that used for its complementary set. A number of parameter values are selected from points along each of these curves, creating a set of simulation parameters for each parameter of interest. An illustration of this sampling approach can be seen in Marino et al [7]. The authors note that due to the symmetrical properties of sinusoidal functions, it is possible that the same parameter value sets could be selected. To address this, a re-sampling scheme is encouraged where a phase shift is introduced into each frequency, and sampling repeated using a slightly different curve [7]. Selection of the number of re-sample curves and parameter values chosen from the curves is an important aspect of running this algorithm, and it is suggested that the user makes themselves familiar with equations provided in Marino et al [7] that aid this decision. Consequently, a number of parameter value sets are created for each parameter of interest, for a particular curve. This process is repeated for an extra parameter, the ‘dummy’, which has an arbitrary value range but no impact on simulation behaviour. This enables a comparison between the impact of each parameter and one known to have no effect on simulation response. As an example, for 7 parameters, plus a dummy, three re-sample curves, and 65 parameter values from points along the curves, 1,560 sets of parameters would be produced. Spartan produces a csv file for each parameter of interest and each curve, containing the parameter value sets on which simulations should be executed. Thus in this example, twenty-four files would be produced, each containing sixty five sets of parameters. For analyses where a large number of parameters are explored, this technique could be computationally expensive [17], [18].

Simulation results are analysed taking into account the frequencies that were used to generate the parameter set used. Through Fourier analysis using these frequencies, variation in output can be partitioned between the parameters, giving an indication of the impact each has on simulation response. Using the equations given in Marino et al [7], two sensitivity indexes are calculated for each parameter-response pairing: a first-order (Si) and total order sensitivity (STi) index. The first indicates the fraction of output variance in that response that can be explained by the value assigned to the parameter. The latter indicates the variance in that response caused by higher-order non-linear effects between the parameter and the others under investigation. Although the dummy parameter has no influence on simulation dynamics, the algorithm may assign this parameter non-zero sensitivity indexes due to aliasing or interference effects, or simplifying assumptions the technique makes in calculating each STi index [7]. Reasoning for this is given in Supplemental material that accompanies the description of the technique in Marino et al [7]. A determination of whether a parameter has a significant impact on simulation response is made by examining the sensitivity indexes, contrasting these with the indexes calculate for the ‘Dummy’ parameter. As this is contrasting sensitivity indexes rather than simulation responses, a statistical measure is generated using a two-sample t-test. Spartan produces both a csv file with these statistics and a plot for each simulation response, detailing the Si and STi indexes for each parameter.

The example simulation data, from which the results presented here were generated, is taken from our recently published lymphoid tissue development simulator (http://www.cs.york.ac.uk/immunesims/frontiers) [9], [10]. An agent-based simulation was developed that captures an abstraction of the early stages of the biological process (12 hours into a 72 hour period). The simulation responses are cell velocity and displacement measures taken at various time-points. All output data and tutorials that reproduce each result described are available at www.cs.york.ac.uk/spartan.

In our published analyses [9], [10] and examples presented here, we focus on six parameters for which a value remains unknown, each constrained to the following respective range: chemoThreshold (0–1), chemoLowerLinearAdjust (0.015–0.08), chemoUpperLinearAdjust (0.1–0.5), thresholdBindProbability (0–1), vcamSlope (0.25–2), and maxVCAMeffectProbabilityCutoff (0–1).

To determine the number of simulation runs required to obtain a representative result, we analysed sample sizes of 1, 5, 50, 100, 300, and 500 runs. Parameter values were kept constant at their calibrated values. Each sample size is analysed in turn using the procedure described, with the generation of 20 subsets of each sample size. This analysis thus required 19,120 individual runs. The online tutorial examines the first five sample sizes. Spartan produces the plots in Figure 1, where Figures 1(a,b,c) show the A-Test scores for all simulation output responses in each of the 20 result sets, for 5, 50, and 100 samples respectively. Figure 1(d) shows the maximum A Test score for each simulation response over the 20 result sets, for all sample sizes analysed. The latter indicates that reducing the effect magnitude of aleatory uncertainty on simulation results to less than ‘small’ (the desired level) requires more than 300 samples when compiling results, thus a sample size of 500 was chosen. It is important to note however that this is specific to our simulation, and unlikely to apply in all cases where spartan is applied. A full analysis for this simulation is found in Patel et al [9]; the online tutorial and results in Figure 1 stop at 300 runs to ensure the tutorial data is of manageable size for download.

In our case study [9], we analysed the six simulation parameters for which biological values are currently unknown using the robustness analysis. The online tutorial demonstrates how both the parameter samples and results described in this section have been generated for two of these parameters.

In this analysis we sought to identify any compound effects that become apparent when the values of the six parameters examined in Technique 2 above are perturbed simultaneously. This has revealed the parameters that are highly influential on simulation behaviour, and provided unique biological insight into the factors that are important at this stage of tissue development. The online tutorial demonstrates how both the parameter samples and results described in this section have been generated.

In this analysis we examined the same six parameters as above, and determined the proportion of variation in simulation response that can be explained by perturbing the value of each parameter. Through use of the eFAST approach [7], [16] we have determined how sensitive the simulation is to each parameter, and thus suggested the impact of each biological factor on tissue development. The online tutorial demonstrates how both the parameter samples and results described in this section have been generated.