AppsCode
  • KubeDB

    Run Production-Grade Databases on Kubernetes

    Stash

    Backup and Recovery Solution for Kubernetes

    KubeVault

    Run Production-Grade Vault on Kubernetes

    Voyager

    Secure Ingress Controller for Kubernetes

    ConfigSyncer

    Kubernetes Configuration Syncer

    Guard

    Kubernetes Authentication WebHook Server

    KubeDB

    KubeDB simplifies Provisioning, Upgrading, Scaling, Volume Expansion, Monitor, Backup, Restore for various Databases in Kubernetes on any Public & Private Cloud

    • check-boxLower administrative burden
    • check-boxNative Kubernetes Support
    • check-boxPerformance
    • check-boxAvailability and durability
    • check-boxManageability
    • check-boxCost-effectiveness
    • check-boxSecurity
    Stash

    A complete Kubernetes native disaster recovery solution for backup and restore your volumes and databases in Kubernetes on any public and private clouds.

    • check-boxDeclarative API
    • check-boxBackup Kubernetes Volumes
    • check-boxBackup Database
    • check-boxMultiple Storage Support
    • check-boxDeduplication
    • check-boxData Encryption
    • check-boxVolume Snapshot
    • check-boxPolicy Based Backup
    KubeVault

    KubeVault is a Git-Ops ready, production-grade solution for deploying and configuring Hashicorp's Vault on Kubernetes.

    • check-boxVault Kubernetes Deployment
    • check-boxAuto Initialization & Unsealing
    • check-boxVault Backup & Restore
    • check-boxConsume KubeVault Secrets with CSI
    • check-boxManage DB Users Privileges
    • check-boxStorage Backend
    • check-boxAuthentication Method
    • check-boxDatabase Secret Engine
    Voyager

    Secure Ingress Controller for Kubernetes

    • check-boxHTTP & TCP
    • check-boxSSL
    • check-boxPlatform support
    • check-boxHAProxy
    • check-boxPrometheus
    • check-boxLet's Encrypt
    configsyncer

    Kubernetes Configuration Syncer

    • check-boxConfiguration Syncer
    Guard

    Kubernetes Authentication WebHook Server

    • check-boxIdentity Providers
    • check-boxCLI
    • check-boxRBAC
  • RESOURCES
  • Webinar New
CONTACT SALES
KubeDB
github-icon twitterx-icon
TRY NOW FREE

New to KubeDB? Please start here.

Reconfigure Druid Cluster

This guide will show you how to use KubeDB Ops-manager operator to reconfigure a Druid Topology cluster.

Before You Begin

  • At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster.

  • Install KubeDB Provisioner and Ops-manager operator in your cluster following the steps here.

  • You should be familiar with the following KubeDB concepts:

To keep everything isolated, we are going to use a separate namespace called demo throughout this tutorial.

$ kubectl create ns demo
namespace/demo created

Note: YAML files used in this tutorial are stored in /docs/guides/druid/reconfigure/yamls directory of kubedb/docs repository.

Now, we are going to deploy a Druid cluster using a supported version by KubeDB operator. Then we are going to apply DruidOpsRequest to reconfigure its configuration.

Prepare Druid Cluster

Now, we are going to deploy a Druid topology cluster with version 28.0.1.

Create External Dependency (Deep Storage)

Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. Amazon S3, Google Cloud Storage, or Azure Blob Storage, S3-compatible storage (like Minio), or HDFS are generally convenient options for deep storage.

In this tutorial, we will run a minio-server as deep storage in our local kind cluster using minio-operator and create a bucket named druid in it, which the deployed druid database will use.


$ helm repo add minio https://operator.min.io/
$ helm repo update minio
$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1

$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \
--set tenant.pools[0].servers=1 \
--set tenant.pools[0].volumesPerServer=1 \
--set tenant.pools[0].size=1Gi \
--set tenant.certificate.requestAutoCert=false \
--set tenant.buckets[0].name="druid" \
--set tenant.pools[0].name="default"

Now we need to create a Secret named deep-storage-config. It contains the necessary connection information using which the druid database will connect to the deep storage.

apiVersion: v1
kind: Secret
metadata:
  name: deep-storage-config
  namespace: demo
stringData:
  druid.storage.type: "s3"
  druid.storage.bucket: "druid"
  druid.storage.baseKey: "druid/segments"
  druid.s3.accessKey: "minio"
  druid.s3.secretKey: "minio123"
  druid.s3.protocol: "http"
  druid.s3.enablePathStyleAccess: "true"
  druid.s3.endpoint.signingRegion: "us-east-1"
  druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/"

Let’s create the deep-storage-config Secret shown above:

$ kubectl create -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/guides/druid/restart/yamls/deep-storage-config.yaml
secret/deep-storage-config created

Now, lets go ahead and create a druid database.

apiVersion: kubedb.com/v1alpha2
kind: Druid
metadata:
  name: druid-cluster
  namespace: demo
spec:
  version: 28.0.1
  deepStorage:
    type: s3
    configSecret:
      name: deep-storage-config
  topology:
    routers:
      replicas: 1
  deletionPolicy: Delete

Let’s create the Druid CR we have shown above,

$ kubectl create -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/guides/druid/update-version/yamls/druid-cluster.yaml
druid.kubedb.com/druid-cluster created

Reconfigure using config secret

Say we want to change the default maximum number of tasks the MiddleManager can accept. Let’s create the middleManagers.properties file with our desire configurations.

middleManagers.properties:

druid.worker.capacity=5

historicals.properties:

druid.processing.numThreads=3

Then, we will create a new secret with this configuration file.

apiVersion: v1
kind: Secret
metadata:
  name: config-secret
  namespace: demo
stringData:
  middleManagers.properties: |-
    druid.worker.capacity=5    
  historicals.properties: |-
    druid.processing.numThreads=3    

$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/guides/druid/update-version/yamls/config-secret.yaml
secret/new-config created

Check Current Configuration

Before creating the druidOpsRequest, first Lets exec into one of the druid middleManagers pod that we have created and check the default configuration:

Exec into the Druid middleManagers:

$ kubectl exec -it -n demo druid-cluster-middleManagers-0 -- bash
Defaulted container "druid" out of: druid, init-druid (init)
bash-5.1$

Now, execute the following commands to see the configurations:

bash-5.1$ cat conf/druid/cluster/data/middleManager/runtime.properties | grep druid.worker.capacity
druid.worker.capacity=2

Here, we can see that our given configuration is applied to the Druid cluster for all brokers.

Now, lets exec into one of the druid historicals pod that we have created and check the configurations are applied or not:

Exec into the Druid historicals:

$ kubectl exec -it -n demo druid-cluster-historicals-0 -- bash
Defaulted container "druid" out of: druid, init-druid (init)
bash-5.1$

Now, execute the following commands to see the metadata storage directory:

bash-5.1$ cat conf/druid/cluster/data/historical/runtime.properties | grep druid.processing.numThreads 
druid.processing.numThreads=2

Here, we can see that our given configuration is applied to the historicals.

Check Configuration from Druid UI

You can also see the configuration changes from the druid ui. For that, follow the following steps:

First port-forward the port 8888 to local machine:

$ kubectl port-forward -n demo svc/druid-cluster-routers 8888
Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888

Now hit the http://localhost:8888 from any browser, and you will be prompted to provide the credential of the druid database. By following the steps discussed below, you can get the credential generated by the KubeDB operator for your Druid database.

Connection information:

  • Username:

    $ kubectl get secret -n demo druid-cluster-auth -o jsonpath='{.data.username}' | base64 -d
admin

  • Password:

    $ kubectl get secret -n demo druid-cluster-auth -o jsonpath='{.data.password}' | base64 -d
LzJtVRX5E8MorFaf

After providing the credentials correctly, you should be able to access the web console like shown below.

  lifecycle

You can see that there are 2 task slots reflecting with the configuration druid.worker.capacity=2.

Create DruidOpsRequest

Now, we will use this secret to replace the previous secret using a DruidOpsRequest CR. The DruidOpsRequest yaml is given below,

apiVersion: ops.kubedb.com/v1alpha1
kind: DruidOpsRequest
metadata:
  name: reconfigure-drops
  namespace: demo
spec:
  type: Reconfigure
  databaseRef:
    name: druid-cluster
  configuration:
    configSecret:
      name: new-config

Here,

  • spec.databaseRef.name specifies that we are reconfiguring druid-prod database.
  • spec.type specifies that we are performing Reconfigure on our database.
  • spec.configSecret.name specifies the name of the new secret.

Let’s create the DruidOpsRequest CR we have shown above,

$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/guides/druid/reconfigure/yamls/reconfigure-druid-ops.yaml
druidopsrequest.ops.kubedb.com/reconfigure-drops created

Check new configuration

If everything goes well, KubeDB Ops-manager operator will update the configSecret of Druid object.

Let’s wait for DruidOpsRequest to be Successful. Run the following command to watch DruidOpsRequest CR,

$ kubectl get druidopsrequests -n demo 
NAME                TYPE          STATUS       AGE
reconfigure-drops   Reconfigure   Successful   4m55s

We can see from the above output that the DruidOpsRequest has succeeded. If we describe the DruidOpsRequest we will get an overview of the steps that were followed to reconfigure the database.

$ kubectl describe druidopsrequest -n demo reconfigure-drops
Name:         reconfigure-drops
Namespace:    demo
Labels:       <none>
Annotations:  <none>
API Version:  ops.kubedb.com/v1alpha1
Kind:         DruidOpsRequest
Metadata:
  Creation Timestamp:  2024-08-02T05:08:37Z
  Generation:          1
  Resource Version:    332491
  UID:                 b6e8cb1b-d29f-445e-bb01-60d29012c7eb
Spec:
  Apply:  IfReady
  Configuration:
    Config Secret:
      Name:  new-kf-topology-custom-config
  Database Ref:
    Name:   druid-prod
  Timeout:  5m
  Type:     Reconfigure
Status:
  Conditions:
    Last Transition Time:  2024-08-02T05:08:37Z
    Message:               Druid ops-request has started to reconfigure druid nodes
    Observed Generation:   1
    Reason:                Reconfigure
    Status:                True
    Type:                  Reconfigure
    Last Transition Time:  2024-08-02T05:08:45Z
    Message:               check reconcile; ConditionStatus:False
    Observed Generation:   1
    Status:                False
    Type:                  CheckReconcile
    Last Transition Time:  2024-08-02T05:09:42Z
    Message:               successfully reconciled the Druid with new configure
    Observed Generation:   1
    Reason:                UpdatePetSets
    Status:                True
    Type:                  UpdatePetSets
    Last Transition Time:  2024-08-02T05:09:47Z
    Message:               get pod; ConditionStatus:True; PodName:druid-prod-historicals-0
    Observed Generation:   1
    Status:                True
    Type:                  GetPod--druid-prod-historicals-0
    Last Transition Time:  2024-08-02T05:09:47Z
    Message:               evict pod; ConditionStatus:True; PodName:druid-prod-historicals-0
    Observed Generation:   1
    Status:                True
    Type:                  EvictPod--druid-prod-historicals-0
    Last Transition Time:  2024-08-02T05:10:02Z
    Message:               check pod running; ConditionStatus:True; PodName:druid-prod-historicals-0
    Observed Generation:   1
    Status:                True
    Type:                  CheckPodRunning--druid-prod-historicals-0
    Last Transition Time:  2024-08-02T05:10:07Z
    Message:               get pod; ConditionStatus:True; PodName:druid-prod-historicals-1
    Observed Generation:   1
    Status:                True
    Type:                  GetPod--druid-prod-historicals-1
    Last Transition Time:  2024-08-02T05:10:07Z
    Message:               evict pod; ConditionStatus:True; PodName:druid-prod-historicals-1
    Observed Generation:   1
    Status:                True
    Type:                  EvictPod--druid-prod-historicals-1
    Last Transition Time:  2024-08-02T05:10:22Z
    Message:               check pod running; ConditionStatus:True; PodName:druid-prod-historicals-1
    Observed Generation:   1
    Status:                True
    Type:                  CheckPodRunning--druid-prod-historicals-1
    Last Transition Time:  2024-08-02T05:10:27Z
    Message:               get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0
    Observed Generation:   1
    Status:                True
    Type:                  GetPod--druid-prod-middleManagers-0
    Last Transition Time:  2024-08-02T05:10:27Z
    Message:               evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0
    Observed Generation:   1
    Status:                True
    Type:                  EvictPod--druid-prod-middleManagers-0
    Last Transition Time:  2024-08-02T05:11:12Z
    Message:               check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-0
    Observed Generation:   1
    Status:                True
    Type:                  CheckPodRunning--druid-prod-middleManagers-0
    Last Transition Time:  2024-08-02T05:11:17Z
    Message:               get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1
    Observed Generation:   1
    Status:                True
    Type:                  GetPod--druid-prod-middleManagers-1
    Last Transition Time:  2024-08-02T05:11:17Z
    Message:               evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1
    Observed Generation:   1
    Status:                True
    Type:                  EvictPod--druid-prod-middleManagers-1
    Last Transition Time:  2024-08-02T05:11:32Z
    Message:               check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-1
    Observed Generation:   1
    Status:                True
    Type:                  CheckPodRunning--druid-prod-middleManagers-1
    Last Transition Time:  2024-08-02T05:11:37Z
    Message:               Successfully restarted all nodes
    Observed Generation:   1
    Reason:                RestartNodes
    Status:                True
    Type:                  RestartNodes
    Last Transition Time:  2024-08-02T05:11:39Z
    Message:               Successfully completed reconfigure druid
    Observed Generation:   1
    Reason:                Successful
    Status:                True
    Type:                  Successful
  Observed Generation:     1
  Phase:                   Successful
Events:
  Type     Reason                                                                     Age    From                         Message
  ----     ------                                                                     ----   ----                         -------
  Normal   Starting                                                                   3m7s   KubeDB Ops-manager Operator  Start processing for DruidOpsRequest: demo/reconfigure-drops
  Normal   Starting                                                                   3m7s   KubeDB Ops-manager Operator  Pausing Druid databse: demo/druid-prod
  Normal   Successful                                                                 3m7s   KubeDB Ops-manager Operator  Successfully paused Druid database: demo/druid-prod for DruidOpsRequest: reconfigure-drops
  Warning  check reconcile; ConditionStatus:False                                     2m59s  KubeDB Ops-manager Operator  check reconcile; ConditionStatus:False
  Normal   UpdatePetSets                                                              2m2s   KubeDB Ops-manager Operator  successfully reconciled the Druid with new configure
  Warning  get pod; ConditionStatus:True; PodName:druid-prod-historicals-0             117s   KubeDB Ops-manager Operator  get pod; ConditionStatus:True; PodName:druid-prod-historicals-0
  Warning  evict pod; ConditionStatus:True; PodName:druid-prod-historicals-0           117s   KubeDB Ops-manager Operator  evict pod; ConditionStatus:True; PodName:druid-prod-historicals-0
  Warning  check pod running; ConditionStatus:False; PodName:druid-prod-historicals-0  112s   KubeDB Ops-manager Operator  check pod running; ConditionStatus:False; PodName:druid-prod-historicals-0
  Warning  check pod running; ConditionStatus:True; PodName:druid-prod-historicals-0   102s   KubeDB Ops-manager Operator  check pod running; ConditionStatus:True; PodName:druid-prod-historicals-0
  Warning  get pod; ConditionStatus:True; PodName:druid-prod-historicals-1             97s    KubeDB Ops-manager Operator  get pod; ConditionStatus:True; PodName:druid-prod-historicals-1
  Warning  evict pod; ConditionStatus:True; PodName:druid-prod-historicals-1           97s    KubeDB Ops-manager Operator  evict pod; ConditionStatus:True; PodName:druid-prod-historicals-1
  Warning  check pod running; ConditionStatus:False; PodName:druid-prod-historicals-1  92s    KubeDB Ops-manager Operator  check pod running; ConditionStatus:False; PodName:druid-prod-historicals-1
  Warning  check pod running; ConditionStatus:True; PodName:druid-prod-historicals-1   82s    KubeDB Ops-manager Operator  check pod running; ConditionStatus:True; PodName:druid-prod-historicals-1
  Warning  get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0                 77s    KubeDB Ops-manager Operator  get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0
  Warning  evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0               77s    KubeDB Ops-manager Operator  evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0
  Warning  check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-0      72s    KubeDB Ops-manager Operator  check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-0
  Warning  check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-0       32s    KubeDB Ops-manager Operator  check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-0
  Warning  get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1                 27s    KubeDB Ops-manager Operator  get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1
  Warning  evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1               27s    KubeDB Ops-manager Operator  evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1
  Warning  check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-1      22s    KubeDB Ops-manager Operator  check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-1
  Warning  check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-1       12s    KubeDB Ops-manager Operator  check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-1
  Normal   RestartNodes                                                               7s     KubeDB Ops-manager Operator  Successfully restarted all nodes
  Normal   Starting                                                                   5s     KubeDB Ops-manager Operator  Resuming Druid database: demo/druid-prod
  Normal   Successful                                                                 5s     KubeDB Ops-manager Operator  Successfully resumed Druid database: demo/druid-prod for DruidOpsRequest: reconfigure-drops

Now let’s exec one of the instance and run a druid-configs.sh command to check the new configuration we have provided.

$ kubectl exec -it -n demo druid-prod-middleManagers-0 -- druid-configs.sh --bootstrap-server localhost:9092 --command-config /opt/druid/config/clientauth.properties --describe --entity-type middleManagerss --all | grep 'log.retention.hours'
  log.retention.hours=125 sensitive=false synonyms={STATIC_BROKER_CONFIG:log.retention.hours=125, DEFAULT_CONFIG:log.retention.hours=168}
  log.retention.hours=125 sensitive=false synonyms={STATIC_BROKER_CONFIG:log.retention.hours=125, DEFAULT_CONFIG:log.retention.hours=168}

As we can see from the configuration of ready druid, the value of log.retention.hours has been changed from 100 to 125. So the reconfiguration of the cluster is successful.

Reconfigure using apply config

Now we will reconfigure this cluster again to set log.retention.hours to 150. This time we won’t use a new secret. We will use the applyConfig field of the DruidOpsRequest. This will merge the new config in the existing secret.

Create DruidOpsRequest

Now, we will use the new configuration in the applyConfig field in the DruidOpsRequest CR. The DruidOpsRequest yaml is given below,

apiVersion: ops.kubedb.com/v1alpha1
kind: DruidOpsRequest
metadata:
  name: kfops-reconfigure-apply-topology
  namespace: demo
spec:
  type: Reconfigure
  databaseRef:
    name: druid-prod
  configuration:
    applyConfig:
      middleManagers.properties: |-
        log.retention.hours=150        
      historicals.properties: |-
        historicals.quorum.election.timeout.ms=4000
        historicals.quorum.fetch.timeout.ms=5000        
  timeout: 5m
  apply: IfReady

Here,

  • spec.databaseRef.name specifies that we are reconfiguring druid-prod cluster.
  • spec.type specifies that we are performing Reconfigure on druid.
  • spec.configuration.applyConfig specifies the new configuration that will be merged in the existing secret.

Let’s create the DruidOpsRequest CR we have shown above,

$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/examples/druid/reconfigure/druid-reconfigure-apply-topology.yaml
druidopsrequest.ops.kubedb.com/kfops-reconfigure-apply-topology created

Verify new configuration

If everything goes well, KubeDB Ops-manager operator will merge this new config with the existing configuration.

Let’s wait for DruidOpsRequest to be Successful. Run the following command to watch DruidOpsRequest CR,

$ kubectl get druidopsrequests -n demo kfops-reconfigure-apply-topology 
NAME                               TYPE          STATUS       AGE
kfops-reconfigure-apply-topology   Reconfigure   Successful   55s

We can see from the above output that the DruidOpsRequest has succeeded. If we describe the DruidOpsRequest we will get an overview of the steps that were followed to reconfigure the cluster.

$ kubectl describe druidopsrequest -n demo kfops-reconfigure-apply-topology 
Name:         kfops-reconfigure-apply-topology
Namespace:    demo
Labels:       <none>
Annotations:  <none>
API Version:  ops.kubedb.com/v1alpha1
Kind:         DruidOpsRequest
Metadata:
  Creation Timestamp:  2024-08-02T05:14:42Z
  Generation:          1
  Resource Version:    332996
  UID:                 551d2c92-9431-47a7-a699-8f8115131b49
Spec:
  Apply:  IfReady
  Configuration:
    Apply Config:
      middleManagers.properties:      log.retention.hours=150
      historicals.properties:  historicals.quorum.election.timeout.ms=4000
historicals.quorum.fetch.timeout.ms=5000
  Database Ref:
    Name:   druid-prod
  Timeout:  5m
  Type:     Reconfigure
Status:
  Conditions:
    Last Transition Time:  2024-08-02T05:14:42Z
    Message:               Druid ops-request has started to reconfigure druid nodes
    Observed Generation:   1
    Reason:                Reconfigure
    Status:                True
    Type:                  Reconfigure
    Last Transition Time:  2024-08-02T05:14:45Z
    Message:               Successfully prepared user provided custom config secret
    Observed Generation:   1
    Reason:                PrepareCustomConfig
    Status:                True
    Type:                  PrepareCustomConfig
    Last Transition Time:  2024-08-02T05:14:52Z
    Message:               successfully reconciled the Druid with new configure
    Observed Generation:   1
    Reason:                UpdatePetSets
    Status:                True
    Type:                  UpdatePetSets
    Last Transition Time:  2024-08-02T05:14:57Z
    Message:               get pod; ConditionStatus:True; PodName:druid-prod-historicals-0
    Observed Generation:   1
    Status:                True
    Type:                  GetPod--druid-prod-historicals-0
    Last Transition Time:  2024-08-02T05:14:57Z
    Message:               evict pod; ConditionStatus:True; PodName:druid-prod-historicals-0
    Observed Generation:   1
    Status:                True
    Type:                  EvictPod--druid-prod-historicals-0
    Last Transition Time:  2024-08-02T05:15:07Z
    Message:               check pod running; ConditionStatus:True; PodName:druid-prod-historicals-0
    Observed Generation:   1
    Status:                True
    Type:                  CheckPodRunning--druid-prod-historicals-0
    Last Transition Time:  2024-08-02T05:15:12Z
    Message:               get pod; ConditionStatus:True; PodName:druid-prod-historicals-1
    Observed Generation:   1
    Status:                True
    Type:                  GetPod--druid-prod-historicals-1
    Last Transition Time:  2024-08-02T05:15:12Z
    Message:               evict pod; ConditionStatus:True; PodName:druid-prod-historicals-1
    Observed Generation:   1
    Status:                True
    Type:                  EvictPod--druid-prod-historicals-1
    Last Transition Time:  2024-08-02T05:15:27Z
    Message:               check pod running; ConditionStatus:True; PodName:druid-prod-historicals-1
    Observed Generation:   1
    Status:                True
    Type:                  CheckPodRunning--druid-prod-historicals-1
    Last Transition Time:  2024-08-02T05:15:32Z
    Message:               get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0
    Observed Generation:   1
    Status:                True
    Type:                  GetPod--druid-prod-middleManagers-0
    Last Transition Time:  2024-08-02T05:15:32Z
    Message:               evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0
    Observed Generation:   1
    Status:                True
    Type:                  EvictPod--druid-prod-middleManagers-0
    Last Transition Time:  2024-08-02T05:16:07Z
    Message:               check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-0
    Observed Generation:   1
    Status:                True
    Type:                  CheckPodRunning--druid-prod-middleManagers-0
    Last Transition Time:  2024-08-02T05:16:12Z
    Message:               get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1
    Observed Generation:   1
    Status:                True
    Type:                  GetPod--druid-prod-middleManagers-1
    Last Transition Time:  2024-08-02T05:16:12Z
    Message:               evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1
    Observed Generation:   1
    Status:                True
    Type:                  EvictPod--druid-prod-middleManagers-1
    Last Transition Time:  2024-08-02T05:16:27Z
    Message:               check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-1
    Observed Generation:   1
    Status:                True
    Type:                  CheckPodRunning--druid-prod-middleManagers-1
    Last Transition Time:  2024-08-02T05:16:32Z
    Message:               Successfully restarted all nodes
    Observed Generation:   1
    Reason:                RestartNodes
    Status:                True
    Type:                  RestartNodes
    Last Transition Time:  2024-08-02T05:16:35Z
    Message:               Successfully completed reconfigure druid
    Observed Generation:   1
    Reason:                Successful
    Status:                True
    Type:                  Successful
  Observed Generation:     1
  Phase:                   Successful
Events:
  Type     Reason                                                                     Age   From                         Message
  ----     ------                                                                     ----  ----                         -------
  Normal   Starting                                                                   2m6s  KubeDB Ops-manager Operator  Start processing for DruidOpsRequest: demo/kfops-reconfigure-apply-topology
  Normal   Starting                                                                   2m6s  KubeDB Ops-manager Operator  Pausing Druid databse: demo/druid-prod
  Normal   Successful                                                                 2m6s  KubeDB Ops-manager Operator  Successfully paused Druid database: demo/druid-prod for DruidOpsRequest: kfops-reconfigure-apply-topology
  Normal   UpdatePetSets                                                              116s  KubeDB Ops-manager Operator  successfully reconciled the Druid with new configure
  Warning  get pod; ConditionStatus:True; PodName:druid-prod-historicals-0             111s  KubeDB Ops-manager Operator  get pod; ConditionStatus:True; PodName:druid-prod-historicals-0
  Warning  evict pod; ConditionStatus:True; PodName:druid-prod-historicals-0           111s  KubeDB Ops-manager Operator  evict pod; ConditionStatus:True; PodName:druid-prod-historicals-0
  Warning  check pod running; ConditionStatus:False; PodName:druid-prod-historicals-0  106s  KubeDB Ops-manager Operator  check pod running; ConditionStatus:False; PodName:druid-prod-historicals-0
  Warning  check pod running; ConditionStatus:True; PodName:druid-prod-historicals-0   101s  KubeDB Ops-manager Operator  check pod running; ConditionStatus:True; PodName:druid-prod-historicals-0
  Warning  get pod; ConditionStatus:True; PodName:druid-prod-historicals-1             96s   KubeDB Ops-manager Operator  get pod; ConditionStatus:True; PodName:druid-prod-historicals-1
  Warning  evict pod; ConditionStatus:True; PodName:druid-prod-historicals-1           96s   KubeDB Ops-manager Operator  evict pod; ConditionStatus:True; PodName:druid-prod-historicals-1
  Warning  check pod running; ConditionStatus:False; PodName:druid-prod-historicals-1  91s   KubeDB Ops-manager Operator  check pod running; ConditionStatus:False; PodName:druid-prod-historicals-1
  Warning  check pod running; ConditionStatus:True; PodName:druid-prod-historicals-1   81s   KubeDB Ops-manager Operator  check pod running; ConditionStatus:True; PodName:druid-prod-historicals-1
  Warning  get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0                 76s   KubeDB Ops-manager Operator  get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0
  Warning  evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0               76s   KubeDB Ops-manager Operator  evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0
  Warning  check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-0      71s   KubeDB Ops-manager Operator  check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-0
  Warning  check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-0       41s   KubeDB Ops-manager Operator  check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-0
  Warning  get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1                 36s   KubeDB Ops-manager Operator  get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1
  Warning  evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1               36s   KubeDB Ops-manager Operator  evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1
  Warning  check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-1      31s   KubeDB Ops-manager Operator  check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-1
  Warning  check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-1       21s   KubeDB Ops-manager Operator  check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-1
  Normal   RestartNodes                                                               15s   KubeDB Ops-manager Operator  Successfully restarted all nodes
  Normal   Starting                                                                   14s   KubeDB Ops-manager Operator  Resuming Druid database: demo/druid-prod
  Normal   Successful                                                                 14s   KubeDB Ops-manager Operator  Successfully resumed Druid database: demo/druid-prod for DruidOpsRequest: kfops-reconfigure-apply-topology

Lets exec into one of the druid middleManagers pod that have updated and check the new configurations are applied or not:

Exec into the Druid middleManagers:

$ kubectl exec -it -n demo druid-with-config-middleManagers-0 -- bash
Defaulted container "druid" out of: druid, init-druid (init)
bash-5.1$

Now, execute the following commands to see the configurations:

bash-5.1$ cat conf/druid/cluster/data/middleManager/runtime.properties | grep druid.worker.capacity
druid.worker.capacity=5

Here, we can see that our given configuration is applied to the Druid cluster for all brokers.

Now, lets exec into one of the druid historicals pod that have updated and check the new configurations are applied or not:

Exec into the Druid historicals:

$ kubectl exec -it -n demo druid-with-config-historicals-0 -- bash
Defaulted container "druid" out of: druid, init-druid (init)
bash-5.1$

Now, execute the following commands to see the metadata storage directory:

bash-5.1$ cat conf/druid/cluster/data/historical/runtime.properties | grep druid.processing.numThreads 
druid.processing.numThreads=3

Here, we can see that our given configuration is applied to the historicals.

Verify Configuration Change from Druid UI

You can access the UI similarly by doing port-forward as mentioned in Check Configuration from Druid UI

You should be able to see the following changes in the UI:

  lifecycle

You can see that there are 5 task slots reflecting with our provided custom configuration of druid.worker.capacity=5.

Cleaning Up

To clean up the Kubernetes resources created by this tutorial, run:

kubectl delete kf -n demo druid-cluster
kubectl delete druidopsrequest -n demo reconfigure-drops
kubectl delete secret -n demo new-config
kubectl delete ns demo

Next Steps

What's on this Page

Search
Improve This Page