The tradition of socio-spatial neighbourhood change research extends back roughly a century, to the Chicago and Atlanta schools of urban sociology [8–10]. These early efforts pioneered methods of close observation and data collection, including contributing to the modernization of census bureaus and the creation of standardized census tract boundaries to facilitate neighbourhood comparison [11]. Even if they have been criticized and modified over time, many of the key concepts and typologies from this period remain relevant and in circulation today [12]. These include the concentric zone model, in which groups compete for prime central space and change radiates out from there [13], or the neighbourhood succession and filtering model in which neighbourhoods defined by upper status groups are replaced in cycles by lower status groups [14–16].

Mid-century researchers carried this tradition forward with the aid of novel statistical techniques. Shevsky and Bell [17], in particular, elaborated factorial ecology as a research paradigm for multivariate classification of neighbourhood types and change [17]. Factorial ecology utilized factor analysis to reduce many variables to a small number of components, which in turn were used to characterize neighbourhoods and their changes (see [18, 19]). This research suggests that neighbourhoods could be differentiated along three primary dimensions, which summarize their economic, family, and ethnic status. While contemporary research often adds additional factors such as housing, transit, and built form [16], a key legacy of factorial ecology is to incorporate variables describing these dimensions [20–24].

If the multidimensional character of factorial ecology never entirely disappeared from academic research, it fell by the wayside for several decades. Instead, neighbourhood and urban change research tended to focus on single variables, often in the form of segregation indices [25]. Nevertheless, largely in market research, geodemographics emerged in this period and promoted cluster analysis as an alternative approach to detecting neighbourhood types and change as complex combinations of many variables [26, 27]. In particular, geodemographic researchers argued that cluster analysis could detect both major types and important but rare types that tend to be lost in factor analysis.

In recent decades, these geodemographic techniques began to appear more in an academic context [28–30]. A key advance was to perform the cluster analysis on many variables across multiple time points, and then to compare transitions over time across the resulting neighbourhood types [7, 31–33]. While these studies failed to exploit the analytical potentials in the underlying transition matrices, they did generate some important results, primarily with regard to US neighbourhoods: Stability (including persistent poverty) is much more common than change. Polarization over time (declining middle class neighbourhoods) is a general tendency. Neighbourhood succession (downgrading) does not necessarily follow the linear trajectory envisioned by earlier theorists, and varies greatly by city. Upgrading (including gentrification) occurs less frequently than one might imagine from popular discussions and is focalized in ethnically diverse neighbourhoods located in a few large metro areas. Finally, this research emphasized the existence of a great diversity of neighbourhood change types, many of which do not neatly fit into the categories of stability, succession, or upgrading in that they involve movement among neighbourhood types with similar socioeconomic status.

More recent studies of socio-spatial neighbourhood change have advanced in formal sophistication as well as geographic and temporal scope. The work of Delmelle (e.g., [34]) has moved the field forward decisively in these regards. Perhaps most importantly, this has occurred by making neighbourhoods’ longer-term temporal trajectories the primary unit of analysis and adapting sequence analysis techniques from genomics (see also [35]). This work largely confirmed key substantive results from previous US studies regarding the main patterns of change. However, it added by locating temporal pathways in space, finding, for instance, that high poverty minority and wealthy white neighbourhoods expand in a relatively contiguous pattern. In contrast, multiethnic areas grow in a more dispersed pattern.

Current research tends to focus on the nature, patterns, and causes of change itself [36]. Most use some form of cluster analysis, but there are several ongoing debates [37]. Authors often question the merits of prototype-based approaches (i.e. k-means) or hierarchical approaches [25]. There are also discussions about the value of sequence analysis or Markovian approaches. Advocates of sequence analysis [34] note that first-order Markov chains disconnect singular transitions from the broader pattern of which they are a part and cannot distinguish later vs. earlier transitions of the same type (e.g., ABAAAA looks the same as AAAABA). Advocates of Markovian approaches argue that sequence analysis misses the generative processes that produce trajectories and that the results of sequence analysis are highly unstable, as they are very sensitive to the particular algorithms used [38]. Also, Markov chains allow predictions to be made regarding future outcomes based solely on its present state, and such predictions are often just as good as the ones considering the full history of the process [39]. These predictions moreover can be made under counterfactual scenarios, either by changing the transition probabilities or the initial distribution of states, thus enabling the researcher to study possible outcomes of new policies, change of preferences, and the like. One drawback of using Markov chains for this procedure is that they do not allow for the emergence of possible new neighbourhood types.

Finally, a common thread in current research involves efforts to spatialize time and temporalize space, either by mapping temporal processes or by pursuing more formal techniques such as spatial Markov analysis [40] for examining how temporal change shifts across geographic contexts. In fact, geographic contexts have proven to be of significant importance in helping to understand neighbourhood dynamics [41, 42]. In particular, Delmelle et al. [41] investigated whether a neighbourhood’s geographic situation impacts its probability of improvement and decline concerning its quality of life profile. To tackle this dependence question, the authors explore a spatial Markov model [40]. Although traditional Markov models have been explored in different contexts, including urban dynamics [42], they miss capturing spatial influences on the process under study. Delmelle et al. [41] observed that declining or improving quality of life is not spatially independent, i.e., surrounding neighbourhoods have clear impacts on a neighbourhood’s probability of improved or diminished quality of life.

Our study continues and extends these past related works. Like most recent work, we use cluster analysis to classify neighbourhood types and examine transitions through those types. We differ by incorporating the hierarchical (nested) structure of the resulting neighbourhood typology into the analysis. This approach enables us to meaningfully include more clusters than is typical in the literature. Similarly, we carry forward the tradition of studying transition probabilities and examining spatial patterns of change, but add by mapping transition rates themselves (rather than sequences). Likewise, similar to the work noted above, we examine the impact of spatial context through spatial Markov chains. Perhaps most significantly, however, in contrast to past work, we exploit Markov models’ formal properties to evaluate their predictive power based on real data and consider the effects of various counterfactual scenarios. This innovation is carried out in both the spatial and non-spatial cases. Here we highlight concepts such as thresholds and non-linearity that are characteristic of complexity theories of cities.

While socio-spatial neighbourhood change research has often been very empirically focused, primarily through its historical legacy to the Chicago School of urban sociology, it has maintained a theoretical basis that points toward key concepts from complexity thinking. Abbott [43] has stressed this theoretical linkage, noting the importance in this tradition of “turning points” (similar to non-linearity and tipping points), fractal patterns, the emergence of global order from local interaction, and the deep integration of space and time [43].

Many of the insights driving efforts to conceive cities as complex systems derive from the work of Jane Jacobs and Christopher Alexander in the 1960s. The core ideas researchers took from these authors included the notion that behind the seeming disorder and diversity of urban forms, there are strong patterns. These emerge from the myriad local decisions and behaviours by individual actors and organizations [44]. These processes of self-organization suggested treating cities as complex systems. They exhibit emergent orders that require significant energy to sustain, patterns of segregation rooted in competition over space, and punctuated equilibria [45]. In particular, complexity studies of cities have tended to emphasize how self-organization involves three major steps [46]: emergence (where local interactions between parts give rise to complex global structures); steady states (where the system maintains itself recursively); and bifurcation or phase transition (in which the old equilibrium dissolves and a new one emerges).

The major contribution of complexity theories of cities has been primarily theoretical. Researchers have sought to bring diverse urban phenomena, often considered independently, into a common framework [46]. Much work, in particular, has sought to elaborate the mathematical basis for this synthesis. These have joined with considerable use of computer simulation models, especially Agent Based Models and Cellular Automata [44, 47], to demonstrate self-organization and emergence in general and highlight some of their specific urban patterns (such as larger zones of stability and local persistent areas of experimentation and change at their boundaries [48]. At the same time, the sorts of long-term empirical studies of local and multilevel change characteristic of socio-spatial neighbourhood change research are relatively rare (though see [49]).

While much of this work unfolded in a mode of rapid theoretical and methodological growth, more recent studies start from the point of view that complexity theories of cities have “come of age” [50]. They are a relatively mature set of theories and methods, and the main challenges involve clarification, application, and refinement. An area of particular concern involves demonstrating the relevance of complexity ideas to substantive areas of urban studies to avoid the fate of past quantitative research paradigms, which in pursuing narrow technical questions lost sight of larger concerns [51]. Heavy reliance on simulations geared toward short-term predictions similarly makes considering longer-term transformations difficult, leading some researchers to develop more explicitly evolutionary models [52].

A related concern involves challenges in importing assumptions from physics into urban studies, given that cities are composed of human agents that are themselves highly complex cognitive-emotional entities that react to changes in their environments [53]. This has made incorporating complexity theories into urban planning programs difficult, in that complexity stresses a certain degree of unpredictability, non-linearity, and multiple forking paths that are not easily assimilated into clear planning interventions on the model of an engineer pulling levers in a machine [54]. In general, complexity theories caution humility in planning; they also seek to rethink the nature of planning theory to some degree. In particular, some advocate scenario based thinking that considers possible futures under different assumptions, while incorporating sensitivity to the notion that small initial differences can compound into large changes over time [50, 55].

Neighbourhood change research has for its part investigated phenomena characteristic of urban complexity, in particular threshold effects and non-linearity. Galster [56, 57] summarizes much of this work, highlighting key theorized models of the underlying behavioural processes (such as socialization, gaming, tolerance, contagion, and diminishing returns) as well as empirical evidence of non-linear neighbourhood dynamics. This empirical work typically uses regression-based techniques on single variables or indexes to examine whether there are critical thresholds where changes in variables such as poverty rates generate larger changes at later time periods. Overall, past research has found the most striking thresholds at extremes, such as at the top 10% of poverty rates and has suggested policy value in targeted “remedial” interventions for neighbourhoods approaching potentially dangerous tipping points [56].

Our study draws from complexity theories of cities primarily as a heuristic guide toward interpreting the results of our analyses and formulating our questions. More specifically, considering cities as complex systems leads us to treat neighbourhood change as a multilevel process in which transitions are constrained by a vertical hierarchy but also occur horizontally across higher-order boundaries. Similarly, we seek to locate persistent patterns of stability and volatility. Perhaps most importantly, we adopt the scenario-based approach as a way to imagine possible urban futures. Here we formulate our scenarios in the light of complexity theory by considering hypothetical changes in the transition probabilities of neighbourhood types strategically located at the boundaries among its higher order structuring principles. Crucial as well is our emphasis on how small changes can have large consequences that reverberate throughout the system, affecting other parts indirectly. In contrast to prior work reviewed by Galster [56, 57] that primarily used regression-based models, single variables or indexes, and sought to identify remedial forms of intervention, we use Markov models and multivariate classification techniques to envision counterfactual scenarios and identify where small introductions of rare transitions might mitigate or reverse larger city-wide trajectories toward growing polarization.

Our study uses Toronto, Canada, as a case study for examining socio-spatial neighbourhood change within a complex urban setting. Toronto has received significant attention from urban researchers in recent years, in large part because of its rapid growth and dramatic demographic transformations. In 2019, Toronto was the fastest growing city in North America, gaining nearly as many new residents as New York lost [1] and especially since the 1980s international migration from East and South Asia, Africa, the Caribbean, and the middle East have made it one of the world’s most ethnically diverse multicultural cities [2].

Amidst these broader metropolitan changes, scholars have turned their attention to neighbourhood change. The vast majority of this research does not use multivariate geodemographic classification techniques but rather examines neighbourhood trends in single variables or indexes, such as income, occupation, or ethnicity, e.g., [4, 5]. This research has identified a long-term economic decline of the suburban zone relative to a gentrifying core, marked by the growth of young professional and “creative class” enclaves in the latter [58, 59], and growth in lower income non-European immigrant groups in the former [60], often concentrated in its ageing social housing high-rise towers and alongside or nearby older Southern and Eastern European communities. These are generally located in the city’s Northeast and Northwest areas where rapid transit service is relatively poor, whereas older established upper status suburban neighbourhoods generally follow the subway lines.

Still, some research has used multivariate classification approaches to identify neighbourhood types in Toronto, such as Principal Component Analysis [3], Self Organizing Maps [61], or cluster analysis [35]. This and similar univariate work [58] tends to find that a three-part structure has largely persisted even amidst the city’s ongoing transformations: a dense downtown core largely populated by young people and highly educated “creative class” professionals and aspirants; traditional suburban areas with large single family homes often inhabited by members of Canada’s historical Protestant elite, Toronto’s main Jewish areas, as well as areas settled by newer Chinese immigrants; a relatively marginalized inner suburban areas in the Northeast and Northwest populated primarily by lower income non-white residents (in particular South Asian, Black, and Arab) along with older white, often Italian, working class communities. In this core-periphery pattern, Toronto resembles other large post-industrial cities such as London, UK [3].

Our study builds on and extends this multivariate research. By taking a multilevel, hierarchical approach, we incorporate the three-part structure identified by past research as a more general structuring principle organizing the city’s pattern of neighbourhood types. But by nesting more specific types within this structure, we are able to incorporate a richer set of types than past research. Moreover, again in contrast to the few studies that have examined change across multivariate neighbourhood types (e.g. [35]), this nested approach allows us to study how neighbourhoods’ positions within this structure shape their trajectories of change. Finally, no prior studies of multivariate neighbourhood types of which we are aware have used Markov chains to predict neighbourhood trajectories in Toronto, study the spatial dependency of change across neighbourhood types, or as a basis for conducting counterfactual thought experiments to explore possible futures as informed by ideas from complexity theories of cites. By filling this gap, we add an important case to the broader literature on socio-spatial neighbourhood change.

Synthesizing and extending the work reviewed above, this study pursues three interrelated research questions. We (i) extend existing work on geodemographic segmentation to Toronto and identify its main types of neighbourhoods and their organization (spatially and hierarchically); (ii) extend prior use of Markov models by validating their accuracy and using them to examine key patterns of neighbourhood change over time, and their spatial dependency; and (iii) pursue ideas from complexity theory by developing counterfactual experiments to determine how these trends would change under different hypothetical scenarios, paying attention to non-linear dynamics.

We use publicly available data from 1996-2016 for census tracts located within the Toronto Census District (CD) [62] representing five different points in time t ∈ T = {1, 2, 3, 4, 5}. Consistent with past studies [7, 16, 31–33], we classify neighbourhoods according to a broad set of demographic, socioeconomic, and housing characteristics (all variables used is in Table 1).

As census-tracts boundaries can change over time, a fundamental challenge when making a longitudinal comparison of spatial data is to ensure that boundaries are consistent. Our dataset minimizes this problem by utilizing the methodology proposed by Allen and Taylor [63]. This approach harmonizes census-tracts to a common set of boundaries, using map-matching, dasymetric overlays, and population-weighted interpolation. Some census-tracts did not contain data for some variables, and for the purposes of this analysis they were removed from the dataset (2.5% of all tracts). In total, we study 514 census-tracts: c_i ∈ C = {c₁, c₂, …, c₅₁₄}. We represent each census-tract on the n-dimensional space Vc,t={v1c,t,v2c,t,…,vnc,t} defined by all the n = 28 census variables. Next, all variables are normalized by z-score each year to control for different measurement scales and to enable a fair comparison of changes across time. In other words, we create a new vector Fc,t={f1c,t,f2c,t,…,fnc,t}, where each variable fic,t=(vic,t-μc,t)/σc,t, with μ^c,t and σ^c,t being the average and standard deviation of V^c,t, respectively. Thus, F^c,t represents a specific tract c on the census year t. In this way, the normalized variables in F, therefore, represent a relative value compared to all other values in the city in a particular year.

Our workflow is in part inspired by the work of Elizabeth Delmelle [34], regarding the way clustering is used to extract socioeconomic typologies, though it is consistent with other work in the field [34]. Fig 1 summarizes our main methodological steps, explained in the next sections.

In order to create a longitudinal socioeconomic typology for the census-tracts under evaluation, we perform a clustering process on all census-tracts c ∈ C represented by F^c,t observed on the five years studied. Most neighbourhood classification studies use either a prototype-based clustering (typically k-means) or hierarchical clustering [25], with ongoing debates about their relative merits [7]. In our study, we strike a middle ground by combining both approaches. We use a k-means clustering algorithm to partition our data. However, instead of randomly choosing the initial centroids, which makes the results sensitive to these choices, we select these centroids based on hierarchical clustering (using Euclidean distance with Ward linkage criteria) [64]. This approach is also known as hierarchical k-means. This hybrid method has the benefit of producing a stable solution in a computationally efficient way, while also allowing us to examine the general organization of the clusters within the resulting hierarchical data space. This, in turn, can also guide decisions about the number of clusters to use in the analysis based on a dendrogram.

Following the clustering procedure, we interpret the results according to procedures typical in the neighbourhood classification literature. We use radar plots to identify each cluster’s semantic based on their centers’ most prominent variables. Each cluster then receives a unique name u ∈ U that tries to express its latent meaning. While these names are helpful in making sense of the analysis, their subjective nature does not compromise the results—using different names as labels could provide a similar message. Each tract is then assigned one of those labels five times (each one representing a point in time corresponding to a census year). Finally, a sequence for each tract is created, depicting its longitudinal trajectory through the multidimensional neighbourhood typology: longitudinal trajectory tract_i = (u_t=1, u_t=2, u_t=3, u_t=4, u_t=5), for the five census years under study. We employ hierarchical clustering and also t-Distributed Stochastic Neighbor Embedding (t-SNE), which is a tool for dimensionality reduction that is well suited for the visualization of high-dimensional data, as in our case [65]. Specifically, we use hierarchical clustering to identify similar neighbourhood types based on the clusters’ centroids and place them in a higher-order grouping (in this study, the three most distinct sets of neighbourhood types), confirming this result with t-SNE.

Having generated a longitudinal trajectory for each census-tract through the neighbourhood types, we create two types of Markov chain: a first order Markov chain and a spatial Markov chain. These tools enable us to study the evolution of neighbourhood types over time.

A challenge in any clustering task is the number of groups (or clusters) to feature, and there is no simple solution. Many neighbourhood clustering studies use various statistical techniques as guides, such as within-cluster sum of squares or the silhouette method. In neighbourhood research, these tend to suggest a relatively small number of clusters, often between 4 and 7. For this study, we chose to highlight a somewhat larger number, beginning our search in the 10-15 range. This allows us to retain the benefit of multidimensional clustering—the ability to identify latent meanings arising from combinations of factors—while remaining closer to the underlying variables. With a small number of clusters, changes in these variables may not appear.

To select the specific number of clusters for our analysis, we examine the dendrogram represented in Fig 2.

Looking at the clusters in the 10-15 range (height around 50), the dendrogram suggests focusing on 13 or 14 clusters. We qualitatively examined both of these solutions, and found that the 14 cluster solution did not add significant analytical value in the present context and so opted for the more parsimonious 13 cluster solution. In the 14 cluster solution, the predominantly Asian portions of the city’s inner suburbs, which are largely geographically contiguous, are further subdivided, mostly by socioeconomic status. Examining neighbourhoods at this level of specificity could be valuable for further research focused on this segment of the city. At the same time, we situate the 13 clusters within the context of the city’s higher-order structure, three larger groups, as suggested by the dendrogram. Particularly, for this interpretation task, we use hierarchical clustering and also t-SNE, confirming the distinct separability of three groups (see S1 and S2 Figs).

Table 2 summarizes the neighbourhood typology that emerges from the clustering process. Radar plots with complete results are in the (S3 Fig).

Overall, Toronto is characterized by three main types of neighbourhoods with various subtypes:

“Creative city” neighbourhoods tend to be located in the downtown core and resemble the types of neighbourhoods featured in Richard Florida’s “The Rise of the Creative Class” [68]: high density areas populated by mobile, highly educated young people, singles, artists, science and tech workers, with relatively few low income blue collar and service workers. More specific types include “established creative” neighbourhoods with very high concentrations of art & culture workers, high home values, single family homes, few blue collar and service workers, and relatively low density; “mixed creative” areas, located at the fringes of the downtown core, that exhibit more occupational and ethno-racial diversity; and “young urban professional” neighbourhoods featuring very high concentrations of highly educated young professional singles focalized in the city center. Toronto’s predominantly Portuguese neighbourhoods continue to have a foothold in this part of the city.

This typology resembles other neighbourhood classifications of Toronto derived by different methods [3, 58]. These also identify three major clusters of neighbourhoods with similar geographical, socioeconomic, and demographic patterns. Our typology adds specificity by identifying subtypes within these major patterns and allowing us to examine the complex multilevel interactions between the higher-order structure of the city and its lower-level constituents.

Having identified the structure and semantic of the city’s neighbourhoods, we turn to our second research question. To answer this question, predictions are made based on the Markov chain generated from the data describing transition probabilities among the 13 neighbourhood types listed in Table 2, as described in the methodology. To validate the performance of the Markov model, we first used it to predict the distribution of these thirteen types at the last (fifth) time point for which we have data. This fifth time period was hidden on purpose in the Markov chain construction so we could access the quality of our predictions. As Table 3 shows, we find very low errors in all the experiments, with consistently lower RMSE values than the baseline models (the only exception is predictions based on the identity matrix from year 4). These results indicate that, despite the relatively few time points, our model is robust to predict the distribution of neighbourhood types in the near-term future. After these validation steps, we used the Markov chain generated from all five time points (years 1-5) to generate predictions 50 time steps in the future to examine how the city would evolve if these trends continue and as part of our counterfactual scenarios.

Fig 3 depicts the transition matrix as a network (the full matrix is in the S4 Fig). As in previous neighbourhood change studies, the dominant pattern is continuity. Most of the time, neighbourhoods remain similar from one year to the next: all have at least a .7 probability of reproducing themselves, and most (9/13) are above .85. In this way, each new iteration of the city is at once the same and subtly different from its predecessor. Dramatic change is rare, but for that reason it stands out all the more when it occurs. This helps to explain why the predictions based on the identity matrix considering year 4 to predict year 5 have a good performance.

The most stable types of neighbourhoods are “elite suburban” and “young urban professional,” while the least stable are mixed (suburban and creative) or European ethnic neighbourhoods (Greek, Portuguese). These latter tend to transition into either “mixed” or “established creative” neighbourhoods, while “mixed creative” neighbourhoods, in turn, are relatively unlikely to sustain themselves. Instead, they are highly likely (.15) to transition into “established creative” types. If we think of this pathway as gentrification, then the most likely targets are older European immigrant neighbourhoods. Nevertheless, in virtue of their somewhat interstitial position (they are located in the Western and Eastern transitional zones between the “creative city,” “suburban,” and “marginalized areas”) these neighbourhoods are subject to a diversity of transitions. For example, Greek predominant neighbourhoods tend to transition into “South Asian predominant,” “Chinese predominant,” and “established creative” areas—spanning all three higher-order types. By contrast, “mixed suburban” neighbourhoods, which also show a high degree of volatility, tend to transition into newer immigrant areas, which show little propensity to gentrify. This overall pattern whereby middle class suburban areas are diminishing while high status suburban and downtown youth areas persist matches other recent studies of Toronto [69]. At the same time, the prevalence of horizontal transitions—“Chinese predominant” with “mixed suburban,” “towers” with “black predominant” illustrate the diversity of forms of change identified in past US studies.

While the city exhibits a high degree of continuity, its degree of structure varies. Rather than assume there is an equally powerful structure at work throughout, an important question concerns how much of a given urban environment is structured at all, and to what degree. We find that much of Toronto is very deeply structured, so that it is highly likely to reproduce itself. In fact, some of the most “creative” parts of the city in terms of who is there and what they are doing—areas in which young, highly educated arts and technology workers predominant—are the most stable. By contrast, other parts of the city exhibit creativity where the urban fabric itself is in a state of transition in which neighbourhood forms themselves rise and fall more rapidly. There are, therefore, at least two types of urban creativity at work here. One appears to thrive within a stable urban context that supports a specific set of groups and activities; in the other, the urban form itself is in a more fluid state of experiment and transformation.

These results reveal a complex hierarchical structure governing urban evolution. The major ordering principles of the city reproduce themselves and maintain the differences between them, even as there is a relatively large amount of movement to and from different local formations within them. The transition probabilities for the higher order principles (creative city, suburbs, marginality) are all above .9. Moreover, as Fig 4 shows, the greatest stability through time tends to be located at the geographic center of these areas. At the same time, amidst these “high fidelity” zones, there are smaller zones of volatility. They are generally located in interstitial areas near the edges of the three higher-order zones, that is, where “creative city” borders “marginalized” and “suburban” areas or “marginalized and “suburban” border one another. This spatio-temporal pattern reveals an important dimension of urban complexity that has been highlighted in computer simulation studies of the self-organization of cities [48]: zones of local instability within a more global spatio-temporal stability, where the areas “in between—the boundaries—are thus the most critical areas in the city for socio-spatial changes (and in society at large).” This description of an artificially generated city is remarkably similar to the one we have drawn from real-world data on Toronto.

To investigate these spatio-temporal dependencies more precisely, we generate a spatial Markov chain according to the methodology described above. Full results are shown in the Supporting Information. Before we proceed, we evaluate the influence of neighbourhood type on transition probabilities on immediate spatial surroundings (our spatial case). For that, we use a test of spatial independence, as described in the methodology section. Our results show that spatial dependence in terms of surrounding neighbourhood type has a statistically significant impact on transition probabilities. According to the χ² test, the system is not independent across space because the test statistic is Q = 2354.1 (p < 0.001, DoF = 854), thus, rejecting the null hypothesis of spatial independence. This means that a state’s transition cannot be assumed to be independent of the neighbourhood type of its neighbours.

Next, we highlight some overall patterns and particularly interesting findings in Fig 5. Given the high degree of spatial clustering, many neighbourhood types are never or rarely nearby, and therefore the number of possible spatially dependent transitions is often low. We, therefore, restrict the analysis to transitions that occur at least ten times, causing an increase or decrease of at least 5% on the reproduction rate (diagonal value) compared to the non-spatial case.

Five key results are depicted in Fig 5:

First, spatial dependence is high in general. Across most neighbourhood types, the reproduction rate shifts substantially depending on its neighbours. This was the case for 10/13 neighbourhood types, as we can observe by looking at the columns of Fig 5. For instance, the neighbourhood type “mixed creative” is influenced in this regard by six different types of neighbours.

Altogether, these findings provide strong support for the view of urban evolution advanced by complexity theories of cities. Simultaneous stability and openness arise from a confluence of top-down and bottom-up processes. Zones of local volatility are globally stable, and patterns of global change contain local stability. The temporality of urban evolution is grounded in space, just as the spatial arrangement of the city emerges from its neighbourhoods’ divergent trajectories through time.

To address our third and final research question, we use the Markov chains described above to predict how the city is likely to evolve in the future, and how this evolution would change under different scenarios. Fig 6 shows the difference between the predicted distribution of neighbourhood types and the actual 2016 distribution in 50 time steps, along with two counterfactual scenarios described in the methodology section.

The non-counterfactual results (0 change, the darkest green dot) in Fig 6 show how the city would evolve if it continued forward according to its current trajectory. The dominant trend would be increased socioeconomic polarization: for example, elite suburban neighbourhoods would increase their share of the overall population of neighbourhoods by around 7.5 percentage points (from 8 to 15.5%), and “young urban professional” and “established creative” areas by around 5 percentage points (from 11 and 7 percent, respectively). All together, these three upper status areas would grow from around 26 to 35% of the city as a whole. At the same time, middle income diverse suburban neighbourhoods would decline (by about 5 percentage points), along with most of the city’s ethnic working and service class communities, as well as its “mixed creative” neighbourhoods. If unchecked, current trends point toward a solidification of the “divided city” [70].

A benefit of our methodological approach is that it allows us to envision alternative scenarios that might change this trajectory towards polarization and division. Three key points stand out in examining the counterfactual scenarios in Fig 6. First, in line with complexity theories, small initial changes can have big effects. In both scenarios, the growth of “young urban professional,” “elite suburban,” and “established creative” areas is substantially reduced. By contrast, the decline in the city’s occupationally and ethnically diverse areas is reduced or stabilized in “mixed creative,” “chinese predominant,” “portuguese predominant,” and “south asian predominant” areas. In some cases, such as predominantly black neighbourhoods, the trend reverses to net growth. Second, we see some signs of non-linear thresholds, again in line with complexity theories of cities. The incremental change from a .01 change in transition probabilities to a .02 change in transition probabilities generates relatively sharp downstream effects, most strikingly in the case of “young urban professional,” “elite suburban,” and “black predominant” neighbourhood types. However, the effects are non-linear and diminish at higher levels. For example, there is very little difference in the effect of a change from 12% vs. 13%. This non-linearity makes sense in the context of these specific scenarios: we are altering transitions that in the non-counterfactual scenario are very rare. Therefore, lower values (e.g. 1% or 2%) represent the initial introduction of a process that rarely occurred previously. As values increase, the process is in place, and additions do not change the situation as much beyond a certain threshold. The bunching in Fig 6 at higher values shows us approximately where this threshold is for the scenarios in this experiment. And third, we see evidence of indirect effects characteristic of complex systems. While we did not make any change to the transition probabilities for “south asian predominant” or “tower” neighbourhoods, their relative footprint in the city grew compared to the non-counterfactual scenario.

All in all, these results show that in a complex dynamic interacting system, small quantitative changes at critical points can potentially make a substantial qualitative difference. Connecting disconnected and divided upper status areas with lower status areas reduces the isolation of these parts of the city, and helps others to retain their foothold. This, in turn, reveals another sign of a complex system: changes in one part reverberate in others.

Fig 7 summarizes the results of the spatial counterfactual scenarios described in the methodology. Here we consider the effects of changing five tracts from “towers” to “mixed suburban,” while leaving other transition probabilities unchanged. In Scenario 1, the five tracts are randomly distributed around the city; in Scenario 2, they are geographically clustered. These situations allow us to examine the impacts of the initial distribution of neighbourhood types on the evolution of their geographic arrangement. For both scenarios, the figure shows where there is a difference of at least 1% in relation to the non-counterfactual case when predicting the distribution of neighbourhood types. More specifically, if in the non-counterfactual case the model predicts a given neighbourhood type will constitute 10% of the total tracts nearby the target type, and in the counterfactual case the distribution is greater than 11% or less than 9%, we colour the appropriate box in the matrix: blue (at least 1% increase) and red (at least 1% decrease). Rows indicate the conditioned neighbourhood type on the spatial Markov over all other ones (columns).

Fig 7 indicates a substantial similarity between the two scenarios, as we would expect. The median value regarding the distribution of all differences between the same transition probabilities in Scenario 1 and Scenario 2 is 0, with a standard deviation of 0.0066. Also, there is no difference bigger than 3% compared to the non-counterfactual case in any scenario. Nevertheless, some nuances emerge on closer inspection. For instance, while the growth of “mixed suburban” in both scenarios is highly similar, Scenario 1 (Fig 7-left) induces declining shares of “towers” and “black predominant” neighbourhoods in the geographic context of “south asian predominant,” with all the averages statistically different under a 95% confidence interval. In addition, the reproduction rate of “black predominant” is higher on Scenario 1 (average of 0.192 with + −0.008 95% CI) compared to Scenario 2 (average of 0.206 with + −0.002 95% CI). In both scenarios, we note a slight tendency toward a potential form of gentrification. For example, we observe a considerable increase in the probability of a neighbourhood of type “established creative” having a neighbour of type “mixed creative.”

These results suggest that introducing a small change in the initial spatial distribution of neighbourhood types can lead to overall urban transformation, and this tends to be enhanced if these changes are more widely spread than concentrated in one geographic area.