The genotype-phenotype map we use is a modification of the one described in [17]. The evolutionary task set for the gene regulation module is to turn an exponentially decaying morphogen gradient (M) across a field of cells in an embryo into a sharp step function profile of a downstream transcription factor T with its transition at the mid-point of the embryo, as shown in Fig 1. This is accomplished by having the morphogen and an RNA Polymerase R bind to two adjacent non-overlapping binding sites in the cis-regulatory region (C) region of the transcription factor, the promoter P, and a single binding site B adjacent to it; transcription occurs whenever the polymerase binds to the promoter, although both proteins can bind to both binding sites dependent on their binding affinities. Binding to the regulatory region is cooperative due to stabilising interactions between the two proteins when bound at the two adjacent sites. The sequences of M and R at the DNA binding sites are represented by binary strings of length ℓ_pd = 10. The corresponding DNA binding sites B and P are also represented by binary strings of the same length. Interactions between a pair of proteins are similarly represented by binary strings of length ℓ_pp = 5. We assume an exponential morphogen concentration profile [M](x, α), as a function of the position of embryonic cells, x; the decay rate of the morphogen α is represented as a continuous variable, with a relative probability of mutation corresponding to an effective string of length ℓ_α = 10 bases. This results in a genome G, of total length ℓ_G = 60. Protein-DNA and protein-protein binding strengths are determined by the number of mismatches between corresponding strings on the two interacting molecules, where for protein-DNA binding the cost of a mismatch is ϵ_pd = 2k_BT and for protein-protein interactions ϵ_pp = 1k_BT, where k_BT is Boltzmann’s constant multiplied by room temperature (298K). We assume that there is a fixed concentration [R] of polymerase, in each cell. We then follow [28] and assume that the concentration of the transcription factor in a cell at position x ([T](x)) is simply proportional to the probability of the polymerase being bound to the promoter, where this is calculated using standard methods of equilibrium statistical mechanics allowing for all configurations of protein species bound at these two binding sites, as well as none being bound (see Methods for details). The fitness contribution F of the overall patterning phenotype ranges from 0 to κ_F depending on how well expression of the transcription factor is confined to the anterior half of the embryo, as shown in Fig 1 (bottom left), where κ_F is a measure of the relative contribution of this trait to the fitness of the organism. We define a population-scaled fitness contribution 2N_eκ_F, where N_e is the effective population size; for 2N_e κ_F < 1 the effects of selection are weak, and are conversely strong when 2Nκ_F > 1. We also assume that there is a boundary at F = F*, below which the organism is unviable. We simulate evolution as continuous time Markov process. After evolving a single population for a given number of generations, we form two replicates of the population that evolve independently, representing the process of allopatric speciation. At various time points following this imposed isolation, we consider the fitness and viability of various outcrossings between the two populations. A DMI occurs when the fitness contribution of a particular hybrid drops below F*.

The properties of a similar genotype-phenotype map have been previously explored [17]. An important property of this genotype-phenotype map is that only a single mechanism of patterning is found, in which the polymerase (R) binds with intermediate affinity to the promoter (P) but with high affinity to the morphogen (M), while the morphogen binds to the morphogen binding site (B) only above a critical morphogen concentration. This results in a spatial switch once the morphogen falls below this concentration; evolution then fine tunes the relationship between the protein-DNA binding energies, the protein-protein binding energy and the steepness of the morphogen gradient α to turn off transcription at the mid-point of the embryo. Despite a single global solution there are many different combinations of the protein-DNA and protein-protein binding energies and α that give good patterning, and each of these correspond to many possible genotypes (G). Of the different possible binding energies, we find that E M B , E R P , E ˜ R M (binding energy of M to B, R to P, and R to M, respectively, calculated by the equivalent of Eq 3 in the Methods) are under strong selection, whilst the other possible binding energies are essentially neutral with weak selective effects. At large population sizes it is found that the evolutionary dynamics exhibits what is known as quenched-disorder in statistical physics, where energy phenotypes that are less constrained take different random values between independent evolutionary runs with no further substitutions; this indicates an underlying roughness to the fitness landscape and that these weakly selected traits are trapped in a local optimum [17].

A key property determining the rate at which incompatibilities arise is the distribution of common ancestor phenotypes as a function of the population-scaled fitness contribution 2N_e κ_F, as shown in Fig 2. For a given value of κ_F, we see that for large population sizes (2Nκ_F ≫ 10) the distribution is what we would expect from conventional evolutionary theory on a fitness landscape with a fitness maximum. In contrast, as the population size is decreased, we find the distribution shifts to lower fitness values to the point when selection is weak (2Nκ_F ≤ 1) the distribution is poised at the inviability boundary. This effect arises due to genetic drift at low population sizes pushing populations towards marginally fit phenotypes that correspond to the largest number of genotypes, that is, with the largest sequence entropy.

Our genome is composed of 4 loci: 1) the R locus corresponding to the polymerase sequence, 2) the Morphogen (M) locus, 3) the C locus which corresponds to the sequences for the cis-regulatory region of the transcription factor and 4) the α locus, which is the morphogen gradient steepness α. Hybrids between the two lineages are constructed by independent reassortment of these loci assuming complete linkage within each locus and no linkage between them. We define a hybrid genotype by a 4 letter string where each letter corresponds to one of the loci defined above and takes one of two cases correspond to whether the allele is from the 1st line or 2nd line; for example, the hybrid rMCa corresponds to R locus having an allele from the 1st lineage, M locus with the allele from the 2nd lineage, the transcription factor (cis-regulatory) C locus from the 2nd lineage and α locus the allele from the 1st lineage. Note that the underlying sequence of each hybrid changes as different substitutions are accepted in each lineage; the notation only refers to alleles fixed at any point in time. We can represent all combinations of the four loci drawn from the two parents (RMCA, RMCa, RMcA, etc.) as points on a four-dimensional Boolean hypercube. In total there are 2⁴ − 2 = 14 hybrids.

In Fig 3, we plot a typical time series of how the fitness of two different hybrids (Rmca(a & b) and RMcA(c & d) changes over a divergence time μt separating a pair of lineages, for 2Nκ_F = 1 (a & c) and 2Nκ_F = 10 (b & d), where μ = ℓ_Gμ₀ is the mutation rate for all base pairs in all loci. 2Nκ_F > 1 indicates strong selection, whilst 2Nκ_F ≤ 1 indicates weak selection where genetic drift dominates (For reference, in human populations it has been estimated that ≈ 20 − 30% of mutations are weakly selected [29, 30], compared to in Drosophila < 10% [30].). We see that the fitness of hybrids generally decreases in a stochastic fashion; when the log-fitness of a hybrid drops below the threshold F* (indicated by the dashed line), a DMI arises as is indicated by a vertical log-fitness line (F = −∞) for that hybrid. As can be seen in Fig 3, at any given time a changing subset of the fourteen possible hybrids might be incompatible.

One important and well explored example is the evolution of transcription factor DNA binding [15, 16, 22, 26, 27, 49, 50], where the genotype-phenotype map from sequence to binding affinity can be explicitly enumerated under simplifying assumptions [51, 52]. These investigations show that for small populations dominated by genetic drift, evolution does not optimise fitness. Rather, there is a trade off between the high fitness of a small number of sequences that bind well and the exponentially larger number of sequences that bind less well. The result is the maximisation of a combination of fitness and the number of sequences that correspond to that phenotype. We can take advantage of analogies with statistical mechanics and represent this combination as the “free fitness”, where the log of the number of sequences is the “sequence entropy” of a phenotype [15, 26, 27, 53]. In this formulation, the effective population size is analogous to an inverse temperature for a physical system connected to a heat-bath, where decreasing population size increases the effect of drift and the importance of sequence entropy relative to fitness.

When the free fitness framework is applied to the role of transcription factor DNA binding in allopatric speciation, our previous work gave rise to a simple prediction: incompatibilities arise more quickly for smaller, drift-dominated, populations [26, 27], supporting previous computational studies by Tulchinsky et al. [54], that showed decreased hybrid fitness for smaller populations. This can be understood as a result of the greater importance of sequence entropy for small populations, resulting in common ancestors with a higher drift load, which are therefore closer to incompatible regions. As a result, fewer substitutions are required for the development of hybrid incompatibilities [26, 27]. Conversely, those transcription factor binding site pairs under weaker selection, at a fixed population size, will give rise to incompatibilities more quickly, as they are more susceptible to drift and in the common ancestor will have a larger drift load.

In this paper, we examine speciation in a more realistic genotype phenotype map. For the first time we examine how incompatibilities arise in allopatry for a simple evolutionary model of developmental system drift, where a higher level organismal spatial patterning phenotype is maintained by stabilising selection, whilst the underlying molecular binding energy phenotypes and ultimately the sequences that determine them, the genotype, are allowed to drift in the evolutionary simulations. Earlier analyses of this model demonstrated the evolution of a number of non-trivial features such as a balance between fitness and sequence entropy deciding the course of evolution at small population sizes and a roughness to the fitness landscape for phenotypes which have high fitness [17]. Importantly here, unlike in previous works [26, 27] we do not directly select for high binding affinity, but only on the organismal level phenotype, but as we discuss, we find the same population size dependence, as well as a number of other novel phenomenon to the speciation process, which would not be obtainable by modelling selection only at the level of phenotypes or genotypes. The results show that biophysics and effective population size provide a much stronger constraint than previous simple modelling of the dynamics of hybrid incompatibilities would suggest [5, 31].

A key result we find is that small populations are characterised by a power law growth of incompatibilities with time, vs large populations a diffusive law (discussed below). Thus we suggest that empirical evidence of power law growth in incompatibilities is a signature of allopatric speciation at small population sizes. The Orr model of the growth of hybrid incompatibilities predicts that incompatibilities grow as a power law of the divergence time between allopatric lineages [5, 31], where the exponent represents the number of genes participating in the interaction (e.g. 2 for a 2-way incompatibility). The results of our model also yield this prediction, but only when populations sizes are sufficiently small. There is, however, an alternative model for the power law behaviour to the combinatoric argument made by Orr. As argued in [27], at small population sizes, where genetic drift is dominant and there is a large drift load, common ancestor populations are poised close to the incompatibility boundary and the growth of DMIs at short times is determined by the likelihood that a few critical substitutions arrive quickly, which is given by a Poisson process; if the critical number of substitutions is K* then for short times we would expect P I ( t ) ∼ ( μ t ) K * and so given that at least n substitutions are needed for a n-way incompatibility, we would expect K* ≥ n. In this paper, we introduced a new method to decompose DMIs into their fundamental pair-wise, 3- and 4-way incompatibilities, and find that for more complex incompatibilities (more loci involved) the larger the exponent of their power law growth. However, we find the exponents we measure for 3- and 4-way incompatibilities are smaller than the predicted exponents of 3 and 4 respectively. We suggest this could be due, as shown in the S1 Text, to the greater number of higher order DMIs arising just by chance, leading to an overestimation of 3- and 4-way DMIs at short times, where at short times a smaller exponent corresponds to a larger number (i.e. τⁿ⁻¹ > τⁿ for τ < 1, where τ is some dimensionless timescale).

Examining the growth of incompatibilities at large population sizes, we see there is a characteristic negative curvature on a log-log plot, predicted theoretically by [26], indicating that, as the number of substitutions needed for incompatibilities is large, the changes in the hybrid traits can be approximated by a diffusion process. However, we find that a simple model of diffusion does not fit the simulation data well; instead a model of sub-diffusion, that would arise if there are a number of kinetic traps giving a broad distribution of substitution times, does fit the data well. This is consistent with the finding that the genotype-phenotype map has a rough fitness landscape, which is only revealed at sufficiently large population sizes [17]. However, it is not clear whether we would observe quenched disorder and sub-diffusive behaviour with more realistic biophysical models that include a larger alphabet size with 20 amino acids and 4 nucleotides.

We also find that incompatibilities arise more rapidly in smaller populations, which is an emergent effect due to the genotype-phenotype map, giving a bias in degeneracy of different phenotypes; lower fitness and less sharp patterning organismal phenotypes have many more sequences than higher fitness, sharper patterning, phenotypes. In smaller drift-dominated populations, this means there is bias towards phenotypes of small sequence entropy (log degeneracy) that counteracts the tendency of natural selection to favour phenotypes of high fitness. Consequently, the common ancestor in small populations is more likely to be slightly maladapted and less substitutions are needed before hybrids develop incompatibilities. These predictions are consistent with empirical evidence for an inverse correlation of speciation rates with effective population size; the net rates of diversification from phylogenetic trees [55–57] indicate smaller populations speciate more quickly, as well from inferred times for post-zygotic isolation to arise [58–60], where for example mammals and cichlids, which have effective population sizes of order 10⁴ [61– 64], develop reproductive isolation more quickly than birds, which have effective populations sizes of order 10⁶ [65]. This model and the similar results obtained for transcription factor DNA binding [26, 27] provide a robust explanation of how stabilising selection can give rise to this population size effect in speciation, which do not require passing through fitness valleys as do models based on the founder effect [66–69].

However, the results for this genotype-phenotype map for developmental system drift are particularly noteworthy compared to the previous results on transcription factor DNA binding, as they are obtained without directly selecting for high affinity, low sequence entropy, binding phenotypes; here we only select on the organismal spatial patterning phenotype, but nonetheless we find small populations develop hybrid incompatibilities more quickly through a similar mechanism of the interplay between fitness, sequence entropy and populations size. Although studies with more complex genotype-phenotype maps will be required, we suggest this points towards a broad principle, where the specificity required of a phenotype to be functional and of high fitness will mean that it will be coded by fewer genotypes. For example, in simple models of protein stability, the empirical observation that all proteins tend to be marginally stable, can be explained by the fact that as the stability of a protein is increased the number of sequences that give that stability decreases rapidly [19]; assuming high fitness corresponds to maximum stability, this phenotype is highly specified, as only a few sequences will meet the requirement that all inter-chain interactions in the protein are favourable.

Another property that emerges from this model not obtainable by simply modelling transcription factor DNA binding is that certain molecular phenotypes are more important than others in giving rise to incompatibilities. One particular feature of this model is that the selective constraints on the different molecular binding energy phenotypes emerge through the evolutionary process of stabilising selection on the organismal phenotype, and are not specified by the model. Most strikingly, and counterintuitively, the model predicts that molecular phenotypes that are under the weakest selective constraints (but not strictly neutral) dominate by giving rise to the earliest incompatibilities for intermediate and large population sizes. Remarkably, here this emerges as a consequence of stabilising selection on the organismal phenotype and not due to selection imposed for good binding affinity as in previous works [27].

We note that these results have been obtained by changing the population size whilst keeping the strength of selection on the organismal trait κ_F fixed. It would be tempting to use these results to suggest that overall those traits in a genome under weakest selection would give rise to the earliest incompatibilities and hence dominate allopatric speciation. However, the question of the how the dynamics of hybrid incompatibilities changes as the strength of selection changes is a subtle one, which we leave to future work; in this model a reduction in κ_F has the effect of changing the phenotypic regions of incompatibility, with non-trivial consequences. It should also be noted, the role of sequence entropy in giving a strong population size dependence to the rate of reproductive isolation; if we consider only a peaked phenotypic landscape without a sequence entropy bias, a reduction in population size would only lead to a broadening of the phenotypic distribution and not a change in the mean of the distribution, and so a much weaker effect, as the common ancestor will still be typically far from incompatible regions. On the other hand with strong (exponential) degeneracy biases, the mean phenotype of the common ancestor changes strongly giving the large population size effect seen, which is as demonstrated in Fig 2.

Another finding of significance is that pair-wise or 2-way DMIs dominate compared with higher order DMIs (3- and 4-way in this model with 4 loci). This is in contrast to Orr’s theoretical argument that predicts a very specific ratio of 2-way: 3-way: 4-way DMIs, equal to 12: 24: 14, which assumes that the fraction of viable paths from the common ancestor to the current day species increases as we consider higher order DMIs [5]. This argument partly rests on the assumption that the number of inviable genotypes remains fixed as a larger number of loci are considered, which would seem a very strong assumption. Despite its simplicity, the genotype-phenotype map in this paper has many of the key features required for higher levels of epistasis, with protein-DNA binding, protein-protein binding and control of the morphogen steepness, all interacting in a non-linear fashion to produce a single gene expression patterning phenotype and so there is clearly the potential for the Orr prediction to be verified; in contrast, we find the converse and our results show there is no bias towards 3-way DMIs, in fact showing instead that the ratio of 2-way to 3-way DMIs is at short times many orders of magnitude larger. This suggests, in this simple, but still relatively complex model, that biophysical constraints are far more important than a purely combinatorial argument would suggest. Evidence could be obtained from more detailed studies similar to [6, 70], where a power law with an exponent greater than 2 would indicate higher-order DMIs are dominant; currently this evidence suggests a quadratic growth law, however, a study with more time-points or species-pairs would provide more confidence. An alternative approach would be to look for linkage disequilibrium between unlinked regions of hybrid genomes, such as was found with hybrids of two species of swordtail fish [71], and though computationally challenging and requiring large numbers of parallel datasets, compare this against evidence for pervasiveness of higher order epistasis. Although recent results of [72], would seem to contradict our conclusions, their finding of extensive complex epistasis relates to higher order interactions between sites within a single locus, coding for protein stability or enzymatic activity, whereas our work relates to epistasis between multiple loci. Similarly, the results of hybrid incompatibilities within RNA molecules [73], which show a ‘spiralling complexity’ of DMIs would appear to be of limited biological relevance to allopatric speciation in higher organisms, as these are related to epistasis within a single locus, which are unlikely to segregate into different recombinants in a hybrid.

Finally, for small populations we find clustering in the behaviours of growth of different types of DMIs, in particular, 3-way DMIs (S1 Text), which can be explained by the different sequence entropy constraints on different molecular phenotypes. This degeneracy is then lifted at larger population sizes and each n-way DMI takes on a different identity in their pattern of growth; this has strong analogy to physical phenomenon in statistical physics where constraints of symmetry dominate at large temperatures, in a regime where noise is important, but at smaller temperatures this symmetry is broken.

A clear future direction to investigate would be the effect of multiple transcription factors binding to enhancer regions to control gene expression [74–76] in large gene networks, where there is potential scope for complex epistasis across many loci coding for a large number of transcription factors. However, as our results show, despite the possibility and a prior expectation of a larger number of triplet interactions, pair-wise interactions dominate; for complex transcriptional control, if pair-wise interactions between proteins, and proteins and DNA dominate, for example in determining the binding affinity of transcriptional complexes, then our conclusions would hold. As we broaden the scope to large gene regulatory networks, there is no strong and direct empirical evidence for pervasive higher order epistasis in their function, which could give rise to higher order incompatibilities being dominant [77]. Specifically, although there is evidence that higher order incompatibilities have arisen in natural populations [78–81], nonetheless a survey of these findings suggest there is no evidence for their dominance [81] as would be predicted by Orr and would be consistent with our findings that point towards biophysics providing a stronger constraint.

Overall, our results point to a basic principle, where developmental system drift or cryptic variation [7, 10, 11, 43], play a key role in speciation; basic body plans or phenotypes are conserved, but co-evolution of the components and loci of complicated gene regulatory networks can change differently in different lineages, giving incompatibilities that grow in allopatry. Here, we suggest a universal mechanism, where the rate of growth of incompatibilities is controlled by the drift load, or distribution of phenotypic values, of the common ancestor, which in turn is determined by a balance between selection pushing populations towards phenotypes of higher fitness and genetic drift pushing them towards phenotypes that are more numerous (higher sequence entropy); this basic mechanism would predict in general that traits under weaker selection will dominant the initial development of reproductive isolation. In particular, although in principle more complicated regulation could give rise to more complex patterns of epistasis [5], our findings suggest that more simple, pair-wise, incompatibilities dominate the development of reproductive isolation between allopatric lineages under stabilising selection.

We model the binding energies of proteins to DNA using the “two-state” approximation [51, 52], which assumes that the binding energy of each amino acid-nucleotide interaction at the binding interface is additive and to a good approximation controlled by the number of mismatches, which each have the same penalty in binding affinity. The various protein-DNA binding energies in the main text are then given by the Hamming distance between the respective sequences. We assume these energies E are measured relative the background free energy of specific and non-specific binding all other sites in the genome, such that the probability of a given transcription factor being bound to a single site is p = 1/(1+e^{(E−μ)/k_BT}), where μ is the chemical potential (∼log(concentration)) of the TF [17, 26, 82]. For example, the binding energy between the morphogen (M) and the first binding site (B) is given by E M B = ϵ p d ρ ( g m , g B ) (3) where ρ(g_m, g_B) is the Hamming distance between the protein binding sequence (g_m) for the morphogen and the sequence for a first regulatory binding site (g_B), where ϵ_pd is the cost in energy for each mismatch. We assume ϵ_pd = 2k_BT as a typical value for the mis-match energy, which are found to be in the range 1−3k_BT [51, 52]. Similarly the co-operative protein-protein binding energy, for example between RNAP and the morphogen is E ˜ R M is E ˜ R M = ϵ p p ( ℓ p p - ρ ( g R , g M ˘ ) ) (4) where g_R is the sequence involved in protein-protein interactions for the polymerase, and g M ˘ represents the equivalent binary sequence for the morphogen, flipped about its centre, which mimics the chirality of real proteins and prevents the co-operative stability from always being maximum for homo-dimers. The parameter ϵ_pp is the stability added for each favourable hydrophobic interaction between amino acids, which we assume to be ϵ_pp = −k_BT. Given ℓ_pp = 5 this gives interactions consistent with typical literature values of −2 to −7k_BT for hydrophobic interactions between proteins [28, 83].

The morphogen concentration profile is approximately exponential; the exact profile we use is [ M ] ( x ) = [ M 0 ] cosh ( α ( x - L ) ) sinh ( α L ) (5) where this arises from solving the reaction-diffusion equation with reflecting boundary conditions and is valid for all α.

To calculate this probability, we use the canonical ensemble of statistical mechanics, for which the partition function Z is most simply expressed in terms of a spin-like variable, which represents the occupation of each binding site, σ j = { 0 , R , M }, Z = ∑ σ P ∑ σ B e - ( E σ P P + E σ B B - μ σ P - μ σ B + E ˜ σ P σ B ) / k B T with E 0 j = 0, E ˜ i i ′ = 0, for either i = 0 or i′ = 0 and μ₀ = 0, where μ σ j = k B T ln [ σ j ] represents the chemical potentials of the protein species at the j^th binding site with [σ_j] being the concentration of species σ_j and 0 represents a free binding site. Formally this construction is known as a 3−state Pott’s model. So given a ‘genome’ G = [g_R, g_r, g_M, g_m g_P, g_B] from which the protein-DNA and protein-protein binding energies are calculated (E_ij and E ˜ i i ′, respectively), p_RP is given schematically by pRP=p(σB−R↱)=1Z((0−R↱)+(M−R↱)+(R−R↱)) (6) where, for example, ( M - R ↱ ) ≡ e - ( E R P + E M B - μ R - μ M + E ˜ R M ) / k B T is the Boltzmann factor for co-operative binding of the morphogen and RNAP to R. Note that we ignore in the partition sum protein-protein binding when not bound to DNA, since these co-operative binding energies tend to be relatively weak compared to DNA binding.

We use a kinetic Monte Carlo scheme to simulate the evolutionary process for the genome G and α on two independent lineages, assuming a fixed effective population size of N, and that we are in the regime of small effective population size (ℓ_G μ₀N ≪ 1, where μ₀ is the base-pair mutation rate). This means the population is represented by a single fixed sequence (or number for α) for all of the loci at each time-point, where effectively mutations are either instantly fixed or eliminated. Specifically, we use the Gillespie algorithm [84], to simulate evolution as a continuous time Markov process; at each step of the simulation the rate of fixation of all one-step mutations from the currently fixed alleles (wild type) is calculated, and one of these mutations is selected randomly in proportion to their relative rate. Time is then progressed by K⁻¹ ln(u), where K is the sum of the rates of all one-step mutants and u is a random number drawn independently between 0 and 1, which ensures the times at which substitutions occur is Poisson distributed, as we would be expected for a random substitution process. The rates are based upon the Kimura probability of fixation [85]: k = μ 0 N 1 - e - 2 δ F 1 - e - 2 N δ F ≈ μ 0 2 N δ F 1 - e - 2 N δ F , (7) where δF is the change of fitness of a mutation at a particular location, given by fitness function defined in the main text, and μ₀N is the rate at which mutations arise in the population; the latter approximation in Eq 7 assumes δF ≪ 1. Note that although in the simulations we use the full form for the fixation probability, fitness effects are typically small (δF ≪ 1) in the simulations, so the substitution rates only depend on the population-scaled fitness changes 2NδF which, for a given mutation, is proportional to 2Nκ_F. Finally, we allow continuous ‘mutations’ in the morphogen steepness parameter α, chosen from a Gaussian distribution of standard deviation δα = 0.5 and assign it an 10 effective base-pairs, which are used when assigning relative weight in the kinetic Monte Carlo scheme, where the total number of base-pairs is ℓ_G = 60.

We determine the Malthusian or log fitness of the spatial gene regulation, from the resulting concentration profile [TF](x) by use of a functional that promotes expression of theTF in the anterior half, whilst penalising expression in the posterior half, with truncation selection below a critical value F*: F = { κ F ln ( W ) if κ F ln ( W ) > F * - ∞ if κ F ln ( W ) < F * (8) where, W [ [ T F ] ( x ) ] = ∫ 0 L / 2 [ T F ] ( x ) d x - ∫ L / 2 L [ T F ] ( x ) d x L 2 max x { [ T F ] ( x ) } . (9) where we use a value of F* = −1.6 × 10⁻³, which corresponds a value of W ≈ 0 . 2, when κ_F = 10⁻³. Strictly, an inviability on a lineage or a hybrid should correspond to W = 0 or F* = −∞, however, these values were chosen to so that a reasonable number of incompatibilities arise in a simulation; for comparison the typical maximum of the integral W ≈ 0 . 6. In this paper, we explore how the changing the population scaled strength of selection (2Nκ_F) affects the rate of reproductive isolation, by keeping κ_F fixed and varyingN accordingly. Note that although here the exact form of the fitness is slightly different to the one used in [17], the qualitative behaviour is the same (S1 Text).

The speciation simulations consist of two replicate simulations starting with the same common ancestor and with the same fitness function. We draw the common ancestor from the equilibrium distribution for G and α. To do this we start from a random initial genome, and run one long simulation for 100,000 substitutions for a fixed scaled population size 2Nκ_F, in order to effectively equilibrate the system (typically 10,000 substitutions are required to adapt to an ensemble of fit states). This represents a reference equilibrium state; different random draws from the equilibrium distribution then consist of running the simulation for a further 100 substitutions.

Given a pattern of hybrid incompatibilities, for example, as shown in Fig 4, if there is a 2-way DMI (e.g. between C and α loci, which we denote I_Ca), then all four hybrid-genotypes containing this DMI (e.g. RMCa, RmCa, rMCa, rmCa) are inviable; these genotypes define a two-dimensional subspace (or face) of the hypercube. Similarly, the points (e.g. rmcA, RmcA, which we denote I_mcA) containing a 3-way DMI form a one-dimensional subspace (or line), while a 4-way DMI takes up only a single point in the four-dimensional hypercube. These different 2-way, 3-way and 4-way DMIs are the fundamental incompatibility types which we seek to explain the pattern of hybrid inviable genotypes observed, for example, as in Fig 4a.

However, this decomposition is hugely underdetermined, as there are only 2⁴ − 2 = 14 possible hybrids (not including the well-adapted genotypes of line 1 and line 2) and a total of I_max = 3^L + 1 − 2^L+1 = 50, different fundamental incompatibilities, for L = 4 loci. This arises as the total number of n−point DMIs is ( 2 n - 2 ) ( L n ), as there are ( L n ) combinations ofN loci amongst L total loci and then considering a binary choice of alleles across both lines, there are a total of 2ⁿ allelic combinations or states, 2 of which are the fit allelic combinations where all alleles come from one lineage or the other giving 2ⁿ − 2. For example, between each pair of loci there are 2² − 2 = 2 mismatching combinations of alleles (e.g. rM and Rm) that could give DMIs and ( L 2 ) = L ( L - 1 ) / 2 = 6 pairwise interactions. A similar argument would give a total of 24 3-way DMIs as there are 2³ − 2 = 6 mismatching combinations of alleles at 3 loci (e.g., excluding rmc and RMC) and ( L 3 ) = 4 3-way interactions and similarly, 14 ( L 4 ) = 14 for 4-way interactions. In total, the max number of DMIs is I m a x = ∑ n = 2 L ( 2 n - 2 ) ( L n ) = 3 L + 1 - 2 L + 1, which for L = 4 loci is I_max = 50.

The approach we take is to find only those combinations of fundamental DMIs that have the smallest total number that can explain the pattern of hybrid incompatibilities, which from a Bayesian perspective would have the smallest Occam factors [86]; for instance, as shown in Fig 4b the list of 6 incompatible hybrid genotypes rmCa, rMCa,RmCa,RMCa,Rmca,RMca, shown by red crosses, can be explained most parsimoniously by three different minimal combinations of fundamental DMIs, each with only 2 DMIs (see main text).