Fault Tolerance & Recovery

On this page

This page guides you through a simple demonstration of how CockroachDB remains available during, and recovers after, failure. Starting with a 6-node local cluster with the default 3-way replication, you'll run a sample workload, terminate a node to simulate failure, and see how the cluster continues uninterrupted. You'll then leave that node offline for long enough to watch the cluster repair itself by re-replicating missing data to other nodes. You'll then prepare the cluster for 2 simultaneous node failures by increasing to 5-way replication, then take two nodes offline at the same time, and again see how the cluster continues uninterrupted.

Before you begin

Make sure you have already installed CockroachDB.

Step 1. Start a 6-node cluster

1. Use the cockroach start command to start 6 nodes:

   $ cockroach start \
   --insecure \
   --store=fault-node1 \
   --listen-addr=localhost:26257 \
   --http-addr=localhost:8080 \
   --join=localhost:26257,localhost:26258,localhost:26259 \
   --background
   

   $ cockroach start \
   --insecure \
   --store=fault-node2 \
   --listen-addr=localhost:26258 \
   --http-addr=localhost:8081 \
   --join=localhost:26257,localhost:26258,localhost:26259 \
   --background
   

   $ cockroach start \
   --insecure \
   --store=fault-node3 \
   --listen-addr=localhost:26259 \
   --http-addr=localhost:8082 \
   --join=localhost:26257,localhost:26258,localhost:26259 \
   --background
   

   $ cockroach start \
   --insecure \
   --store=fault-node4 \
   --listen-addr=localhost:26260 \
   --http-addr=localhost:8083 \
   --join=localhost:26257,localhost:26258,localhost:26259 \
   --background
   

   $ cockroach start \
   --insecure \
   --store=fault-node5 \
   --listen-addr=localhost:26261 \
   --http-addr=localhost:8084 \
   --join=localhost:26257,localhost:26258,localhost:26259 \
   --background
   

   $ cockroach start \
   --insecure \
   --store=fault-node6 \
   --listen-addr=localhost:26262 \
   --http-addr=localhost:8085 \
   --join=localhost:26257,localhost:26258,localhost:26259 \
   --background
   

2. Use the cockroach init command to perform a one-time initialization of the cluster:

   $ cockroach init --insecure --host=localhost:26257
   

Step 2. Set up load balancing

In this tutorial, you run a sample workload to simulate multiple client connections. Each node is an equally suitable SQL gateway for the load, but it's always recommended to spread requests evenly across nodes. This section shows how to set up the open-source HAProxy load balancer.

1. Install HAProxy.

   Mac Linux

   If you're on a Mac and use Homebrew, run:

   $ brew install haproxy