Is it possible for any large-scale communication network to resist targeted attacks? The internet was originally designed to withstand targeted (nuclear) attacks [1], and the resilience of the internet has long been part of common wisdom [2]. But 18 years after Albert et. al [3] showed that the topology of the internet’s router network makes it vulnerable to targeted attacks (vs. random faults), the fundamental problem of attack-tolerant network topology remains unsolved. Attack-tolerant topologies are desirable not only for the router network, but for any physical or virtual network where a compromised node puts communication at risk. For example, the network of verified keys in the public key infrastructure underlying secure http [4], or the network of DNS nameservers. The ongoing vulnerability of the internet is evidenced by a long history of censorship and surveillance incidents achieved by means of targeted attacks [5]. In this paper, we present the first theoretical network topology supporting attack-toloerant, point-to-point networked communication, without relying on infinitely transitive trust [6].

Methods for tolerating various kind of faults within networks are an important and ongoing area of research [3, 7, 8]. Adversarial faults, those in which an adversary can target attacks strategically, deserve special attention. Such attacks are both extremely difficult to guard against and often have important social implications. In particular, censorship and surveillance are often achieved by targeting central network locations and either blocking or capturing the information flowing through them. While cryptograpy can provide some protection against surveillance, it is vulnerable to man-in-the-middle attacks [9], and cannot overcome censorship when communication is blocked. In this paper, we instead consider a topological approach. The Internet’s decentralized design was motivated by the need to withstand targeted attacks, such as nuclear strikes [1]. But despite longstanding common wisdom [2], both theoretical results and recent events have demonstrated that the internet is surprisingly vulnerable to attack.

Analysis of the internet’s router network has shown that while it is remarkably resilient against random faults, it is highly susceptible to adversarial faults [3]. These results have been attributed to the heavy-tailed degree distribution of the Internet’s router network [10, 11]. Random failures are highly likely to affect only low-degree nodes, thus having little effect. However, adversarial faults target the few high-degree nodes, and therefore remove a large number of edges with each fault. So while the protocols of the Internet are decentralized, the network structure is somewhat centralized. In other words, the protocols of the Internet do not require centralization, but centralization may still emerge from the sociotechnical processes that create its network structure.

The internet’s vulnerability to censorship and other targeted attacks has been demonstrated by several recent events. In 2008, YouTube suffered a worldwide outage for several hours when a service provider in Pakistan advertised false routing information [12]. The action (known as a black hole attack) was intended to censor YouTube within Pakistan only, but resulted in a worldwide cascading failure when a router misconfiguration allowed the false routing information to propagate outside of Pakistan. This incident exemplifies the type of attack requiring a topological approach. First, the attack was non-technological (a government order), allowing the attacker to bypass any cryptographic or technology-based defenses. Second, the attack originated at a single point of failure (a misconfigured router). Third, the behavior of the compromised component (the router) cascaded through a network (the network of internet routers) because the correct behavior of other components depended on the correct behavior of the single point of failure. And while the action was not an intentional attack against the global internet, the ability of an attacker to succeed without even trying only highlights the internet’s vulnerability to adversarial faults.

As another example, in 2013, the Texas-based email provider Lavabit was ordered to disclose their private SSL keys to the FBI [13]. Rather than complying, Lavabit ceased operations in order to protect their users from surveillance. Once again, the attack was non-technical. And again, the attack was on a single point of failure: Lavabit’s web server and that server’s TLS/SSL keys. In this case, the affected network was the internet’s public key infrastructure. With the private keys, an attacker would be able to intercept and surveil traffic because the issuing certificate authority (and any users trusting that authority) would incorrectly trust that they were communicating with Lavabit. While originally intended as surveillance, this action effectively became an act of censorship. So we see that such vulnerabilities are not limited to any one system or protocol, but result from centralized structure itself.

This paper addresses the need for a theoretial understanding of network and redundancy-based approaches to attack-tolerance. For the purpose of this paper, attack-tolerance refers to the ability of a pair of communicating nodes to detect, with high probability, when a message has been blocked or altered in the presence of adversarial faults. Our primary result is theoretical: an algorithm for constructing highly redundant paths in a particular network topology. While we motivate the need for such network topologies using examples such as the internet router network and webs of trust, we do not propose that our algorithm as a pracitcal solution for any of these examples. Rather, our result is a demonstration that such topological approaches to attack-tolerance are theoretically possible and suggest the importance of further theoretical and applied work.

We consider a setting in which a source node attempts to route a message to a target node, while an adversary attempts to block or intercept the message by compromising a number of intermediate nodes. We also assume that edges represent direct trust, i.e., the belief that an adversary is unlikely to compromise a neighbor, as in the web of trust approach [14, 15]. The web of trust approach typically assumes infinite trust transitivity in that it is possible for one node to have some level of trust for another as long as they are connected by a path of directly trusted edges, regardless of the length of that path. The assumption of infinite transitivity is unrealistic [6], so we instead make a stricter assumption: bounded trust transitivity, that trust transitivity can only be extended over a finite number of edges. Even with this more restrictive assumption, we show how trust can be established between two nodes even when no path of directly trusted edges exists between them.

Under the above assumptions, we show how to evaluate the influence of network structure on attack-tolerance. We next present a structured multipath fault tolerance (SMFT) scheme to extend standard fault tolerance techniques to the case of adversarial faults in networks [16, 17]. The SMFT scheme requires the existence of a concurrent multipath routing (CMR) algorithm [7, 18, 19], to takes advantage of the independence of faults along independent paths. We also present a novel CMR algorithm for the butterfly network topology. The butterfly topology is popular in parallel processing [20] and peer-to-peer [21, 22] applications, due to its regular structure, low degree, and high connectivity.

It is important to note that the butterfly is a highly structured and constrained network topology, very different form those found in social networks and other self-organized networks. The reader may wonder whether it is realistic or useful to assume such control over the network structure. Regardless of implementation difficulty, we argue that attack-tolerance cannot be achieved without the ability to influence network structure. Targeted attacks, by definition, target single points of failure. Attack tolerance can thus be achieved in two ways: 1. preventing failure at those points, or 2. preventing the existence of central points. Because individual points are always vulnerable to coercive, non-technological attacks, the former method is insufficient. We must instead rely on some control over topology to prevent the existence of single points of failure. The difficulty of achieving control over network topology can be mitigated by a number of approaches. Attack-tolerant networks might be sub-components of larger, less-constrained systems. For example, a single centralized server might be replaced by a distributed network of servers, each with different ownership, physical location, and legal jurisdiction, without placing any unrealistic constraints on the clients connecting to those servers. When attack tolerant topologies are nested (as in the case of the butterfly topology), multiple independent sub-components could be merged into larger ones over time. Real-world examples of structured neworks include: overlay networks [21, 22], formal organizations [23], government-regulated cellular networks [24], and call tree notification systems [25]. In general, when the need for attack-tolerance is high enough to warrant investment in infrastructure, topology can be engineered and maintained as infrastructure.

Our main contributions are:

This paper is organized as follows. Section reviews background and related work. Section describes adversarial fault tolerance on structured networks. Section gives background on the butterfly network topology. Section presents our concurrent multipath routing algorithm for the butterfly network. Section discusses the results. And Section concludes.

There has been considerable work on trust-based attack-tolerance techniques in network security, both centralized and decentralized. Centralized approaches such as public key infrastructure (PKI) suffer from a number of vulnerabilities [4], including vulnerability to coercion, which stems largely from the single points of failure inherent to centralization. The well-known and widely-used web of trust approach [14, 15] is a decentralized alternative. In a web of trust, individuals can have direct trust, as well as indirect trust for those trusted by someone directly trusted. Typically, this transitivity is extended to any number of hops, sometimes reducing trust by a multiplicative factor at each hop. Infinite trust transitivity is helpful for establishing a large group of trusted nodes, but unfortunately unrealistic [6]. Our work addresses this limitation by assuming only bounded trust transitivity.

Previous work applying network topology to attack tolerance has focused on authentication, showing that independent paths can reduce an adversary’s ability to impersonate a target [26]. Other work has shown that identifying independent paths in arbitrary networks is NP-hard and provided approximation algorithms [27]. Our work complements these results by extending our focus beyond authentication, to communication. When network topology can be controlled, we sidestep the NP-hard problem of finding independent paths on arbitrary networks by using the mathematical structure of the butterfly topology to construct provably independent paths.

Many distributed consensus protocols (such as those used by cryptocurrencies) are designed to tolerate arbitrary or adversarial faults. Byzantine agreement protocols [28, 29] provide tolerance against arbitrary faults (including attacks) under some circumstances, but are limited to small networks due to poor scalability. Proof-of-work [30, 31] (blockchain) systems provide better scalability, but are wasteful of computational and energy resources, and do not take advantage of trusted relationships. Federated Byzantine Agreement (FBA) [32] is scalable, allows for flexible trust, and is highly fault-tolerant on networks meeting specific requirements. However, FBA does not provide a method for constructing networks to meet those requirements, or for calculating the failure probabilities within a particular network.

All existing attack-tolerant networks we are aware of are content-addressable networks (CANs) in which data is stored and retrieved based on key values, rather than point-to-point networks, in which data is communicated between two parties. Fiat and Saia described a scheme that combines the butterfly topology with expander graphs to create a highly censorship-resistant, content-addressable network [33], although this scheme requires high levels of data replication and indefinite storage. Perhaps the most mature structural solution is the Freenet collaboration [34]. Freenet uses secret sharing [35, 36] and small-world routing [37, 38] to create a content-addressable network with a high level of both confidentiality and censorship resistance. Freenet guarantees that data is stored redundantly, but still allows for centralized network structure, and thus single points of failure, as data travels from its origin to the redundant storage locations. Unlike the above content-addressable networks, our architecture is purely network based and does not require nodes to store data indefinitely. Our architecture also improves on the scalablity of the Fiat-Saia network, and makes requirements about network topology explicit.

Multipath routing protocols identify multiple paths between source and destination in contrast to traditional unipath routing, which uses a single path. The special case of concurrent multipath routing uses multiple paths simultaneously. Multipath routing has many applications, including reduced congestion, increased throughput, and more reliability [18]. Many of these routing protocols offer increased confidentiality [7]. Some approaches utilize redundant paths as backups for increased fault tolerance [39], and some specifically protect against adversarial faults [40–42]. Most work on multipath routing has been motivated by applications related to wireless sensor networks (WSNs), and have thus focused on ad-hoc, unstructured networks, often having a central base station. The method of Liu et al. [43] routes multiple messages first to random peers and then to a central base station, with the network edges constrained by sensors’ physical location. We have found very few examples of CMR applied to adversarial fault tolerance in the existing literature, and all have focused on ad-hoc wireless sensor networks, without attention to the role of network structure.

Our proposed routing algorithm makes use of a structured network, in which link structure is predetermined. Structured networks have been a popular tool in parallel processing architectures [20]. More recently, peer-to-peer systems based on distributed hash tables have used structured overlay networks to map table keys to local TCP/IP routes [21, 22]. Such networks can be designed to have favorable structural and routing properties, which can be used to to improve attack-tolerance.

Our proposed architecture is differentiated from existing systems by several properties (Table 1). Decentralized architectures are more resistant to coercion [44] and man-in-the-middle attacks [9]. Trust-based systems are more sustainable than proof-of-work. Bounded-trust systems do not require the unrealistic assumption of infinite trust transitivity. Topological approaches address the root cause of vulnerability in heavy-tail networks, rather than relying on technology that can be side-stepped through coercion. Point-to-point communication allows two individuals to exchange messages without requiring large amounts of indefinite data storage on intermediate nodes.

Within the field of fault tolerance, many techniques have been developed for building reliable systems out of unreliable components [16, 17]. We will make use of standard fault tolerance terminology, summarized here. A fault is occurs when one component of a system behaves incorrectly (e.g., a routing node blocks or alters a message). The result of that fault (e.g., a recipient receiving conflicting messages) is an error state. If the error is undetected or corrected to the wrong value, the system has experienced a failure (e.g., an altered message is accepted as authentic). Note that when an error is detected but cannot be corrected, the system has still tolerated the fault because it has not accepted an error state. We are concerned in particular with adversarial faults, which are chosen strategically to maximize the likelihood of a failure.

In order to implement structured multipath fault tolerance, we need a structured network topology with high effective redundancy. In this paper, we apply SMFT to the butterfly network topology [20]. The butterfly network is highly structured, making it most suitable for applications where portions of the network structure can be controlled or influenced. The butterfly network is also recursive, with larger versions composed out of multiple smaller versions, making it possible for independently organized attack-tolerant networks to merge into larger ones over time. More flexible architectures may be possible, but attack-tolerance will always require some level of influence over network structure in order to limit single points of failure. To address the case when the network cannot be fully controlled, we show how partially rewiring a snapshot of the internet’s router network can greatly increase it’s effective redundancy and attack-tolerance properties, without requiring additional edges.

We now present a routing algorithm to construct 2^h independent paths between two nodes in a butterfly network, where h is the trust radius under the partial trust model. Informally, Alice sends each message to a distinct node on her trust boundary, then to a distinct intermediate node in the untrusted region, then to a distinct node on Bob’s trust boundary, and finally to Bob. The intermediate nodes are in a sense “far” from each other and ensure that no two paths overlap in the untrusted region. Each path can be parameterized by a single integer s, which identifies the specific node on Alice’s trust boundary (or equivalently the node on Bob’s trust boundary, or the untrusted intermediate).

The algorithm guarantees paths are independent by ensuring that (outside the trusted neighborhoods) they only include nodes that match the path parameter s at certain indexes in their place-within-level. Since each path has a unique parameter s, its set of untrusted nodes is disjoint from all other paths. As with the unipath routing algorithm, each of the multiple paths proceed from a source v to a destination u using down and down-right edges, cycling through levels one at a time. However, we cycle through the levels twice, once to route from v to a particular path’s intermediary node, and again to route from the intermediary to w. Each cycle is divided into stages, with different properties used to prove independence at each stage (see Fig 7). In the first cycle (stages 1–4), path independence is guaranteed by ensuring that all nodes match the path parameter s in the first h bits of the place-within-level. Similarly, in the second cycle (stages 5–7), independence is guaranteed by ensuring that all paths match s in the h bits of the place-within-level preceding the destination index. A full example is illustrated in Fig 8.

Our work has been motivated by the vulnerability of current communications infrastructure to surveillance and censorship, which are often achieved by coercive targeted attacks against central nodes. We have already discussed two such cases: Pakistan’s inadvertant censorship of YouTube [12] and the FBI’s surveillance and censorship of Lavabit [13].

The reader may wonder how our methods could be employed in scenarios such as large-scale state-sponsored censorship [46]. Censorship-resistant infrastructure often replaces central servers (e.g., the router in the 2008 YouTube incident) with multiple servers across the world, synchronized through consensus protocols. The directory authorities used by the Tor project [47] are one example. However, the size of such networks is limited by the number of trusted relationships (degree) each node can maintain, and the inherent insecurity of extending transitive trust to an ever-larger network. Our work provides both a theoretical framework and a specific example of how network structure can be engineered to leverage trust for a high level of attack-tolerance, without sacrificing scalability.

We have focused primarily on adversarial faults that block or change messages (censorship) but our work is also relevant to surveillance. While cryptographic anti-surveillance techniques exist, they remain vulnerable to man-in-the-middle attacks, in which an intermediate node masquerades as the destination. Such attacks can be detected if the original message reaches the true destination unaltered, which SMFT can help to ensure.

In its current form, our work has several limitations. Most obviously, it requires complete control over the network structure. However, we have shown that even partially control over network structure can improve attack tolerance properties. Still a more flexible network structure is desirable. There is also the question of how to construct such a network without a central authority. This limitation may not be as severe as it seems, due to the nested structure of the butterfly network. We conjecture that smaller independently-formed networks could be merged into a single larger network without central coordination. When nodes and edges exist in geographic space (as in cables connecting internet routers) this scheme might require connections between very distant locations. Such connections are extremely expensive to construct and maintain, although many exist today in the form of redundant internet backbones and undersea cables. An alternative solution might involve satellite links, which connect distant geographic points much more easily. While turning our network theoretical results into practical applications will require considerable additional work, we believe that work is inevitably necessary in order to create attack-tolerant networks.

In addition to addressing the above limitations, we see several potential directions for future work. The development of new structured networks or multipath routing algorithms could achieve higher levels of redundancy and attack-tolerance. It is also desirable to examine how changes to social dynamics could shift self-organized networks towards a more decentralized structure. Finally, our results could be implemented to address specific applications, e.g., secure messaging, domain name resolution, or anonymous web browsing.

Coercion-resistant, topological approaches to attack tolerance are needed to address the current vulnerability of communications infrastructure to censorship and surveillance. We have presented a novel concurrent multipath routing (CMR) algorithm for the butterfly network, as well as a structured multipath fault tolerance (SMFT) scheme, which can be combined to create a coercion-resistant, attack-tolerant point-to-point communication architecture. We have also shown how assuming bounded trust transitivity can enable a quantitative analysis of the relationships between network structure, trust, and attack-tolerance. In our architecture, the probability of an adversary causing an undetectable error decreases exponentially with the network’s effective redundancy. The effective redundancy, in the case of the butterfly topology, grows exponentially with the radius of trust transitivity. Furthermore, a small increase in the number of messages sent (traffic volume) can compensate for a large increase in the number of messages compromised by an adversary. These results require some control over the structure of a network, or some portion of the network. Even when network structure cannot be perfectly controlled, we have shown that partially rewiring a snapshot of the internet’s router network can greatly increase its attack-tolerance properties. We believe that this work provides a foundation for the development of additional topology-based communication architectures to guard against technical and coercive adversarial attacks, including censorship and surveillance.