New to KubeDB? Please start here.

KubeDB - Redis Sentinel

This tutorial will show you how to use KubeDB to provision a Redis Sentinel .

Before You Begin

Before proceeding:

Read redis sentinel concept to learn about Redis Sentinel.
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using kind.
Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps here.
To keep things isolated, this tutorial uses a separate namespace called demo throughout this tutorial. Run the following command to prepare your cluster for this tutorial:
```
$ kubectl create ns demo
namespace/demo created
```

Note: The yaml files used in this tutorial are stored in docs/examples/redis folder in GitHub repository kubedb/docs.

Deploy Redis Sentinel

First RedisSentinel instance needs to be deployed and then a Redis instance in Sentinel mode which will be monitored by the RedisSentinel instance.

The following is an example RedisSentinel object which creates a Sentinel with three replicas.

apiVersion: kubedb.com/v1
kind: RedisSentinel
metadata:
  name: sen-demo
  namespace: demo
spec:
  version: 6.2.14
  replicas: 3
  storageType: Durable
  storage:
    resources:
      requests:
        storage: 1Gi
    storageClassName: "standard"
    accessModes:
    - ReadWriteOnce
  deletionPolicy: WipeOut

$ kubectl create -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/examples/redis/sentinel/sentinel.yaml
redissentinel.kubedb.com/sen-demo created

Here,

spec.replicas denotes the number of replica nodes
spec.storage specifies the StorageClass of PVC dynamically allocated to store data for this database. This storage spec will be passed to the PetSet created by KubeDB operator to run database pods. So, each members will have a pod of this storage configuration. You can specify any StorageClass available in your cluster with appropriate resource requests.

KubeDB operator watches for RedisSentinel objects using Kubernetes API. When a RedisSentinel object is created, KubeDB operator will create a new PetSet and a Service with the matching RedisSentinel object name. KubeDB operator will also create a governing service for PetSets named kubedb, if one is not already present.

Now we will deploy a Redis instance with giving the sentinelRef to the previously created RedisSentinel instance. To deploy a Redis in Sentinel mode, specify spec.mode field in Redis CRD.

The following is an example Redis object which creates a Redis Sentinel with three replica node, and it is monitored by Sentinel instance sentinel

apiVersion: kubedb.com/v1
kind: Redis
metadata:
  name: rd-demo
  namespace: demo
spec:
  version: 6.2.14
  replicas: 3
  sentinelRef:
    name: sen-demo
    namespace: demo
  mode: Sentinel
  storage:
    resources:
      requests:
        storage: 1Gi
    storageClassName: "standard"
    accessModes:
      - ReadWriteOnce
  deletionPolicy: Halt

$ kubectl create -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/examples/redis/sentinel/redis.yaml
redis.kubedb.com/rd-demo created

Here,

spec.mode specifies the mode for Redis. Here we have used Redis to tell the operator that we want to deploy Redis in sentinel mode.
spec.replicas denotes the number of replica nodes
spec.storage specifies the StorageClass of PVC dynamically allocated to store data for this database. This storage spec will be passed to the PetSet created by KubeDB operator to run database pods. So, each members will have a pod of this storage configuration. You can specify any StorageClass available in your cluster with appropriate resource requests.

KubeDB operator watches for Redis objects using Kubernetes API. When a Redis object is created, KubeDB operator will create a new PetSet and a Service with the matching Redis object name. KubeDB operator will also create a governing service for PetSets named kubedb, if one is not already present.

$ kubectl get redissentinel -n demo
NAME       VERSION   STATUS   AGE
sen-demo   6.2.14     Ready    2m39

$ kubectl get redis -n demo
NAME      VERSION   STATUS         AGE
rd-demo   6.2.14     Ready   2m41s

$ kubectl get petset -n demo
NAME       READY   AGE
rd-demo    3/3     86s
sen-demo   3/3     12m


$ kubectl get pvc -n demo
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-rd-demo-0    Bound    pvc-830fb301-512a-4de9-a110-c0ce032fabca   1Gi        RWO            standard       99s
data-rd-demo-1    Bound    pvc-0bc06618-a7ef-42ef-b2a0-4e5563d68df7   1Gi        RWO            standard       93s
data-rd-demo-2    Bound    pvc-99aebc54-c016-4376-a3a3-25f882ae86e7   1Gi        RWO            standard       87s
data-sen-demo-0   Bound    pvc-c55d804e-67e1-431c-92a6-67bdde14f59c   1Gi        RWO            standard       12m
data-sen-demo-1   Bound    pvc-171e7d75-c423-4c7f-aabd-42ce50cd0ff4   1Gi        RWO            standard       12m
data-sen-demo-2   Bound    pvc-2886e192-845b-4b44-89e0-20c2af64ec47   1Gi        RWO            standard       12m


$ kubectl get pv -n demo
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   REASON   AGE
pvc-0bc06618-a7ef-42ef-b2a0-4e5563d68df7   1Gi        RWO            Delete           Bound    demo/data-rd-demo-1    standard                111s
pvc-171e7d75-c423-4c7f-aabd-42ce50cd0ff4   1Gi        RWO            Delete           Bound    demo/data-sen-demo-1   standard                13m
pvc-2886e192-845b-4b44-89e0-20c2af64ec47   1Gi        RWO            Delete           Bound    demo/data-sen-demo-2   standard                12m
pvc-830fb301-512a-4de9-a110-c0ce032fabca   1Gi        RWO            Delete           Bound    demo/data-rd-demo-0    standard                117s
pvc-99aebc54-c016-4376-a3a3-25f882ae86e7   1Gi        RWO            Delete           Bound    demo/data-rd-demo-2    standard                104s
pvc-c55d804e-67e1-431c-92a6-67bdde14f59c   1Gi        RWO            Delete           Bound    demo/data-sen-demo-0   standard                13m


$ kubectl get svc -n demo
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
rd-demo           ClusterIP   10.96.165.208   <none>        6379/TCP    2m40s
rd-demo-pods      ClusterIP   None            <none>        6379/TCP    2m40s
rd-demo-standby   ClusterIP   10.96.193.56    <none>        6379/TCP    2m40s
sen-demo          ClusterIP   10.96.249.99    <none>        26379/TCP   14m
sen-demo-pods     ClusterIP   None            <none>        26379/TCP   14m

KubeDB operator sets the status.phase to Ready once the database is successfully created. status.phase section is similar for Redis object and RedisSentinel object. Run the following command to see the modified RedisSentinel object:

$ kubectl get redissentinel -n demo sen-demo -o yaml

apiVersion: kubedb.com/v1
kind: RedisSentinel
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"kubedb.com/v1","kind":"RedisSentinel","metadata":{"annotations":{},"name":"sen-demo","namespace":"demo"},"spec":{"replicas":3,"storage":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"standard"},"storageType":"Durable","deletionPolicy":"Halt","version":"6.2.14"}}      
  creationTimestamp: "2023-02-03T06:36:16Z"
  finalizers:
  - kubedb.com
  generation: 2
  name: sen-demo
  namespace: demo
  resourceVersion: "531539"
  uid: 9b3785b5-4dc3-47bc-91e2-ba260dabd17e
spec:
  authSecret:
    name: sen-demo-auth
  autoOps: {}
  healthChecker:
    failureThreshold: 1
    periodSeconds: 10
    timeoutSeconds: 10
  podTemplate:
    controller: {}
    metadata: {}
    spec:
      containers:
      - name: redis
        resources:
          limits:
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 1Gi
      serviceAccountName: sen-demo
  replicas: 3
  storage:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
    storageClassName: standard
  storageType: Durable
  deletionPolicy: Halt
  version: 6.2.14
status:
  conditions:
  - lastTransitionTime: "2023-02-03T06:36:16Z"
    message: 'The KubeDB operator has started the provisioning of Redis: demo/sen-demo'
    reason: DatabaseProvisioningStartedSuccessfully
    status: "True"
    type: ProvisioningStarted
  - lastTransitionTime: "2023-02-03T06:36:52Z"
    message: All desired replicas are ready.
    reason: AllReplicasReady
    status: "True"
    type: ReplicaReady
  - lastTransitionTime: "2023-02-03T06:37:13Z"
    message: 'The Sentinel: demo/sen-demo is accepting client requests.'
    observedGeneration: 2
    reason: DatabaseAcceptingConnectionRequest
    status: "True"
    type: AcceptingConnection
  - lastTransitionTime: "2023-02-03T06:37:13Z"
    message: 'The Sentinel: demo/sen-demo is ready.'
    observedGeneration: 2
    reason: ReadinessCheckSucceeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2023-02-03T06:37:20Z"
    message: 'The Redis: demo/sen-demo is successfully provisioned.'
    observedGeneration: 2
    reason: DatabaseSuccessfullyProvisioned
    status: "True"
    type: Provisioned
  observedGeneration: 2
  phase: Ready

Connection Information

Connect to Redis Database

Hostname/address: you can use any of these
- Service: rd-demo.demo
- Pod IP: ($ kubectl get pod -n demo -l app.kubernetes.io/name=redises.kubedb.com -o yaml | grep podIP)
Port: 6379

Username: Run following command to get username,

$ kubectl get secrets -n demo rd-demo-auth -o jsonpath='{.data.\username}' | base64 -d
default

Password: Run the following command to get password,

$ kubectl get secrets -n demo rd-demo-auth -o jsonpath='{.data.\password}' | base64 -d
5VjZ7iYaoo8YRp!p

Now, you can connect to this redis database using the service using the credentials.

Connect to Sentinel

Hostname/address: you can use any of these
- Service: sen-demo.demo
- Pod IP: ($ kubectl get pod -n demo -l app.kubernetes.io/name=redissentinels.kubedb.com -o yaml | grep podIP)
Port: 26379

Username: Run following command to get username,

$ kubectl get secrets -n demo sen-demo-auth -o jsonpath='{.data.\username}' | base64 -d
root

Password: Run the following command to get password,

$ kubectl get secrets -n demo sen-demo-auth -o jsonpath='{.data.\password}' | base64 -d
Gw_sd;~Vrsj9kJSL

Now, you can connect to this sentinel using the service using the credentials.

Check Replication Scenario

# first list the redis pods list
$ kubectl get pods --all-namespaces -o jsonpath='{range.items[*]}{.metadata.name} ---------- {.status.podIP}:6379{"\\n"}{end}' | grep rd-demo
rd-demo-0 ---------- 10.244.0.70:6379
rd-demo-1 ---------- 10.244.0.72:6379
rd-demo-2 ---------- 10.244.0.74:6379

# enter into any pod's container named redis
$ kubectl exec -it -n demo rd-demo-0 -c redis -- bash
/data #

# now inside this container, see which role of this pod
/data #  redis-cli info replication
role:master
connected_slaves:2
slave0:ip=rd-demo-1.rd-demo-pods.demo.svc,port=6379,state=online,offset=1258038,lag=1
slave1:ip=rd-demo-2.rd-demo-pods.demo.svc,port=6379,state=online,offset=1258038,lag=0
master_failover_state:no-failover
master_replid:1ce0d55b8d8c1bd2502d4d7e63b1a2f021dbc938
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:1258038
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset

So, the node rd-demo-0 is master, and it has two connected slaves. If a replica node is being exec, it will show which master it is connected to.

Check Sentinel Monitoring

A sentinel can monitor multiple masters. Sentinel stores information about master and its replicas. Sentinel constantly monitor master and perform failover operation when master fail to respond. Sentinel pings master recurrently after a certain period a time.

$ kubectl get pods --all-namespaces -o jsonpath='{range.items[*]}{.metadata.name} ---------- {.status.podIP}:6379{"\\n"}{end}' | grep sen-demo
sen-demo-0 ---------- 10.244.0.46:6379
sen-demo-1 ---------- 10.244.0.48:6379
sen-demo-2 ---------- 10.244.0.50:6379
# enter into Sentinel pod's container named redissentinel
$ kubectl exec -it -n demo sen-demo-0 -c redissentinel -- bash

# now inside this container, see the masters information which this sentinels monitors
/data #  redis-cli -p 26379 sentinel masters
1)  1) "name"
    2) "demo/rd-demo"
    3) "ip"
    4) "rd-demo-0.rd-demo-pods.demo.svc"
    5) "port"
    6) "6379"
    7) "runid"
    8) "93cc7c6803fb14a87b6cc59d6fffdc96901153e5"
    9) "flags"
   10) "master"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "359"
   19) "last-ping-reply"
   20) "359"
   21) "down-after-milliseconds"
   22) "5000"
   23) "info-refresh"
   24) "8563"
   25) "role-reported"
   26) "master"
   27) "role-reported-time"
   28) "4800628"
   29) "config-epoch"
   30) "0"
   31) "num-slaves"
   32) "2"
   33) "num-other-sentinels"
   34) "2"
   35) "quorum"
   36) "2"
   37) "failover-timeout"
   38) "5000"
   39) "parallel-syncs"
   40) "1"

It can be seen that the master rd-demo-0.rd-demo-pods.demo.svc has two slaves as we deployed Redis with three replicas, and it has two other sentinel instances monitoring it as we have deployed RedisSentinel instance with three replicas as well.

Data Availability

Now, you can connect to this database through redis-cli. In this tutorial, we will insert data, and we will see whether we can get the data from any other node (any master or replica) or not.

Read the comment written for the following commands. They contain the instructions and explanations of the commands.


# connect to any node
$ kubectl exec -it rd-demo-0 -n demo -c redis -- bash
/data #

# now ensure that you are connected to the 1st pod
/data # redis-cli -c -h 10.244.0.140
10.244.0.140:6379>

# set 'world' as value for the key 'hello'
10.244.0.140:6379> set hello world
OK
10.244.0.140:6379> exit

# switch the connection to the replica of the current master and get the data
/data # redis-cli -c -h 10.244.0.72
10.244.0.145:6379> get hello
"world"
# trying to write data in a replica
10.244.0.145:6379> set apps code 
(error) READONLY You can't write against a read only replica.
10.244.0.145:6379> exit

Automatic Failover

To test automatic failover, we will force the master node to sleep for a period. Since the master node (pod) becomes unavailable, sentinel will initiate a failover, so that a replica is promoted to master. When the old master comes back, it will join the cluster as the new replica of the new master.

Read the comment written for the following commands. They contain the instructions and explanations of the commands.

# connect to any node and get the master nodes info
$ kubectl exec -it rd-demo-0 -n demo -c redis -- bash

# Check role of the first pod which has IP 10.244.0.70
/data # redis-cli -h 10.244.0.70 info replication | grep role
role:master

# let's sleep the master node 10.244.0.70 with the `DEBUG SLEEP` command
/data # redis-cli -h 10.244.0.70 debug sleep 120
OK

$ kubectl exec -it rd-demo-0 -n demo -c redis -- bash

# Check role of the first pod which has IP 10.244.0.70
/data # redis-cli -h 10.244.0.70 info replication | grep role
role:slave

# Check role of the second pod which has IP 10.244.0.72
/data # redis-cli -h 10.244.0.72 info replication | grep role
role:master

/data # exit

Notice that 110.244.0.72 is the new master and 10.244.0.70 has become the replica of 10.244.0.72.

Cleaning up

First set termination policy to WipeOut all the things created by KubeDB operator for this Redis instance is deleted. Then delete the redis instance to clean what you created in this tutorial.

$ kubectl patch -n demo rd/rd-demo -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge"
redis.kubedb.com/rd-demo patched

$ kubectl delete rd rd-demo -n demo
redis.kubedb.com "rd-demo" deleted

Now delete the RedisSentinel instance similarly.

$ kubectl patch -n demo redissentinel/sen-demo -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge"
redissentinel.kubedb.com/sen-demo patched

$ kubectl delete redissentinel sen-demo -n demo
redis.kubedb.com "sen-demo" deleted

Next Steps

Replace Sentinel of your database with a new Sentinel
Monitor your Redis database with KubeDB using out-of-the-box Prometheus operator.
Monitor your Redis database with KubeDB using out-of-the-box builtin-Prometheus.
Use private Docker registry to deploy Redis with KubeDB.
Detail concepts of Redis object.
Detail concepts of RedisVersion object.

KubeDB

Stash

KubeVault

Voyager

ConfigSyncer

Guard

RESOURCES

Druid

Elasticsearch

FerretDB

Kafka

MariaDB

Memcached

Microsoft SQL Server

MongoDB

MySQL

OpenSearch

PerconaXtraDB

PgBouncer

Pgpool

PostgreSQL

ProxySQL

RabbitMQ

Redis

SingleStore

Solr

ZooKeeper