New to KubeDB? Please start here.

Kubedb PostgreSQL - Arbiter node for PostgreSQL cluster with even nodes

Here we will show how to use KubeDB to provision a PostgreSQL DB with Arbiter Node.

Before You Begin

At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using kind.

Now,install KubeDB operator in your cluster following the steps here.

Concepts

We use Raft Consensus Algorithm to determine the leader of PostgreSQL cluster. With each postgres container, we run a Sidekick container named pg-coordinator which determins the leader of the cluster both on bootstrap and failover scenarios using raft consensus algorithm. However this works very well with the clusters having odd number of nodes. Clusters with even number of nodes still have a chance of split brain problem(Two primary node in the cluster at the same time). To avoid this split brain issue, we introduced an extra node named {{ db }}-arbiter. The job of the arbiter node is to participate solely in leader election. It doesn’t store any data related to postgresql database. However it needs a bare minimum of 1Gi~2Gi Storage for storing WAL and Snapshots that are generated by RAFT. But the generation of WAL are not very frequent. So storage space of 2Gi is more than enough for most of the clusters.

Demo

To keep things isolated, this tutorial uses a separate namespace called demo throughout this tutorial.

$ kubectl create ns demo
namespace/demo created

Let’s apply the following yaml.

apiVersion: kubedb.com/v1
kind: Postgres
metadata:
  name: ha-postgres
  namespace: demo
spec:
  arbiter:
    resources:
      limits:
        memory: 256Mi
      requests:
        cpu: 200m
        memory: 256Mi
        storage: 1Gi
  replicas: 2
  storageType: Durable
  storage:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
  version: "16.1"

A petset with the name {{ dbName }}-arbiter should be created once you apply this yaml.

kubectl get petset -n demo
NAME                  AGE
ha-postgres           72s
ha-postgres-arbiter   55s

Following a pod having name {{ dbName }}-arbiter-0 will be created.

kubectl get pods -n demo
NAME                    READY   STATUS    RESTARTS   AGE
ha-postgres-0           2/2     Running   0          84s
ha-postgres-1           2/2     Running   0          78s
ha-postgres-arbiter-0   1/1     Running   0          68s

Get the the pvc,

kubectl get pvc -n demo
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-ha-postgres-0           Bound    pvc-8653958e-091c-4c15-8151-4d207c976ad1   1Gi        RWO            standard       2m50s
data-ha-postgres-1           Bound    pvc-afd698cf-22d5-4a66-996e-a8d53198a95f   1Gi        RWO            standard       2m44s
data-ha-postgres-arbiter-0   Bound    pvc-25c413c9-4cb0-4a5c-89f6-9d2208841b07   2Gi        RWO            standard       2m34s

Note: Your pvc size for data nodes might be 1000Gi, but for most of the cases you will never need more than 2Gi storage for arbiter node.

You can check arbiter pod has only one container by getting the pod yaml.

kubectl get pods -n demo ha-postgres-arbiter-0 -oyaml

Why do we need arbiter node?

Few of our users don’t want to run 3 node cluster as this was quite expensive for them, instead they wanted to use 2 node cluster, 1 primary and 1 replica. But due to raft implementation, there was chance of split brain. So we introduced this arbiter node so that they can still have 1 primary and 1 replica cluster and not have faced split brain problem.

Cleaning up

kubectl delete pg -n demo ha-postgres
kubectl delete ns demo

Next Steps

• Learn more about Raft implementation
• Learn about backup and restore PostgreSQL database using Stash.
• Learn about initializing PostgreSQL with Script.
• Learn about custom PostgresVersions.
• Want to setup PostgreSQL cluster? Check how to configure Highly Available PostgreSQL Cluster
• Monitor your PostgreSQL database with KubeDB using built-in Prometheus.
• Monitor your PostgreSQL database with KubeDB using Prometheus operator.
• Detail concepts of Postgres object.
• Use private Docker registry to deploy PostgreSQL with KubeDB.
• Want to hack on KubeDB? Check our contribution guidelines.