New to KubeDB? Please start here.
Druid Topology Volume Expansion
This guide will show you how to use KubeDB
Ops-manager operator to expand the volume of a Druid Topology Cluster.
Before You Begin
At first, you need to have a Kubernetes cluster, and the
kubectl
command-line tool must be configured to communicate with your cluster.You must have a
StorageClass
that supports volume expansion.Install
KubeDB
Provisioner and Ops-manager operator in your cluster following the steps here.You should be familiar with the following
KubeDB
concepts:
To keep everything isolated, we are going to use a separate namespace called demo
throughout this tutorial.
$ kubectl create ns demo
namespace/demo created
Note: The yaml files used in this tutorial are stored in docs/examples/druid folder in GitHub repository kubedb/docs.
Expand Volume of Topology Druid Cluster
Here, we are going to deploy a Druid
topology using a supported version by KubeDB
operator. Then we are going to apply DruidOpsRequest
to expand its volume.
Prepare Druid Topology Cluster
At first verify that your cluster has a storage class, that supports volume expansion. Let’s check,
$ kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 28h
longhorn (default) driver.longhorn.io Delete Immediate true 27h
longhorn-static driver.longhorn.io Delete Immediate true 27h
We can see from the output the longhorn
storage class has ALLOWVOLUMEEXPANSION
field as true. So, this storage class supports volume expansion. We can use it.
Create External Dependency (Deep Storage)
Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. Amazon S3, Google Cloud Storage, or Azure Blob Storage, S3-compatible storage (like Minio), or HDFS are generally convenient options for deep storage.
In this tutorial, we will run a minio-server
as deep storage in our local kind
cluster using minio-operator
and create a bucket named druid
in it, which the deployed druid database will use.
$ helm repo add minio https://operator.min.io/
$ helm repo update minio
$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1
$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \
--set tenant.pools[0].servers=1 \
--set tenant.pools[0].volumesPerServer=1 \
--set tenant.pools[0].size=1Gi \
--set tenant.certificate.requestAutoCert=false \
--set tenant.buckets[0].name="druid" \
--set tenant.pools[0].name="default"
Now we need to create a Secret
named deep-storage-config
. It contains the necessary connection information using which the druid database will connect to the deep storage.
apiVersion: v1
kind: Secret
metadata:
name: deep-storage-config
namespace: demo
stringData:
druid.storage.type: "s3"
druid.storage.bucket: "druid"
druid.storage.baseKey: "druid/segments"
druid.s3.accessKey: "minio"
druid.s3.secretKey: "minio123"
druid.s3.protocol: "http"
druid.s3.enablePathStyleAccess: "true"
druid.s3.endpoint.signingRegion: "us-east-1"
druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/"
Let’s create the deep-storage-config
Secret shown above:
$ kubectl create -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/guides/druid/volume-expansion/yamls/deep-storage-config.yaml
secret/deep-storage-config created
Now, we are going to deploy a Druid
combined cluster with version 28.0.1
.
Deploy Druid
In this section, we are going to deploy a Druid topology cluster for historicals and middleManagers with 1GB volume. Then, in the next section we will expand its volume to 2GB using DruidOpsRequest
CRD. Below is the YAML of the Druid
CR that we are going to create,
apiVersion: kubedb.com/v1alpha2
kind: Druid
metadata:
name: druid-cluster
namespace: demo
spec:
version: 28.0.1
deepStorage:
type: s3
configSecret:
name: deep-storage-config
topology:
historicals:
replicas: 1
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageType: Durable
middleManagers:
replicas: 1
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageType: Durable
routers:
replicas: 1
deletionPolicy: Delete
Let’s create the Druid
CR we have shown above,
$ kubectl create -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/guides/druid/volume-expansion/yamls/druid-topology.yaml
druid.kubedb.com/druid-cluster created
Now, wait until druid-cluster
has status Ready
. i.e,
$ kubectl get dr -n demo -w
NAME TYPE VERSION STATUS AGE
druid-cluster kubedb.com/v1alpha2 28.0.1 Provisioning 0s
druid-cluster kubedb.com/v1alpha2 28.0.1 Provisioning 9s
.
.
druid-cluster kubedb.com/v1alpha2 28.0.1 Ready 3m26s
Let’s check volume size from petset, and from the persistent volume,
$ kubectl get petset -n demo druid-cluster-historicals -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage'
"1Gi"
$ kubectl get petset -n demo druid-cluster-middleManagers -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage'
"1Gi"
$ kubectl get pv -n demo
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-0bf49077-1c7a-4943-bb17-1dffd1626dcd 1Gi RWO Delete Bound demo/druid-cluster-segment-cache-druid-cluster-historicals-0 longhorn <unset> 10m
pvc-59ed4914-53b3-4f18-a6aa-7699c2b738e2 1Gi RWO Delete Bound demo/druid-cluster-base-task-dir-druid-cluster-middlemanagers-0 longhorn <unset> 10m
You can see the petsets have 1GB storage, and the capacity of all the persistent volumes are also 1GB.
We are now ready to apply the DruidOpsRequest
CR to expand the volume of this database.
Volume Expansion
Here, we are going to expand the volume of the druid topology cluster.
Create DruidOpsRequest
In order to expand the volume of the database, we have to create a DruidOpsRequest
CR with our desired volume size. Below is the YAML of the DruidOpsRequest
CR that we are going to create,
apiVersion: ops.kubedb.com/v1alpha1
kind: DruidOpsRequest
metadata:
name: dr-volume-exp
namespace: demo
spec:
type: VolumeExpansion
databaseRef:
name: druid-cluster
volumeExpansion:
historicals: 2Gi
middleManagers: 2Gi
mode: Offline
Here,
spec.databaseRef.name
specifies that we are performing volume expansion operation ondruid-cluster
.spec.type
specifies that we are performingVolumeExpansion
on our database.spec.volumeExpansion.historicals
specifies the desired volume size for historicals node.spec.volumeExpansion.middleManagers
specifies the desired volume size for middleManagers node.spec.volumeExpansion.mode
specifies the desired volume expansion mode(Online
orOffline
).
During Online
VolumeExpansion KubeDB expands volume without pausing database object, it directly updates the underlying PVC. And for Offline
volume expansion, the database is paused. The Pods are deleted and PVC is updated. Then the database Pods are recreated with updated PVC.
If you want to expand the volume of only one node, you can specify the desired volume size for that node only.
Let’s create the DruidOpsRequest
CR we have shown above,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/guides/druid/volume-expansion/yamls/druid-volume-expansion-topology.yaml
druidopsrequest.ops.kubedb.com/dr-volume-exp created
Verify Druid Topology volume expanded successfully
If everything goes well, KubeDB
Ops-manager operator will update the volume size of Druid
object and related PetSets
and Persistent Volumes
.
Let’s wait for DruidOpsRequest
to be Successful
. Run the following command to watch DruidOpsRequest
CR,
$ kubectl get druidopsrequest -n demo
NAME TYPE STATUS AGE
dr-volume-exp VolumeExpansion Successful 3m1s
We can see from the above output that the DruidOpsRequest
has succeeded. If we describe the DruidOpsRequest
we will get an overview of the steps that were followed to expand the volume of druid.
$ kubectl describe druidopsrequest -n demo dr-volume-exp
Name: dr-volume-exp
Namespace: demo
Labels: <none>
Annotations: <none>
API Version: ops.kubedb.com/v1alpha1
Kind: DruidOpsRequest
Metadata:
Creation Timestamp: 2024-10-25T09:22:02Z
Generation: 1
Managed Fields:
API Version: ops.kubedb.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:apply:
f:databaseRef:
f:type:
f:volumeExpansion:
.:
f:historicals:
f:middleManagers:
f:mode:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2024-10-25T09:22:02Z
API Version: ops.kubedb.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
f:observedGeneration:
f:phase:
Manager: kubedb-ops-manager
Operation: Update
Subresource: status
Time: 2024-10-25T09:24:35Z
Resource Version: 221378
UID: 2407cfa7-8d3b-463e-abf7-1910249009bd
Spec:
Apply: IfReady
Database Ref:
Name: druid-cluster
Type: VolumeExpansion
Volume Expansion:
Historicals: 2Gi
Middle Managers: 2Gi
Mode: Offline
Status:
Conditions:
Last Transition Time: 2024-10-25T09:22:02Z
Message: Druid ops-request has started to expand volume of druid nodes.
Observed Generation: 1
Reason: VolumeExpansion
Status: True
Type: VolumeExpansion
Last Transition Time: 2024-10-25T09:22:10Z
Message: get pet set; ConditionStatus:True
Observed Generation: 1
Status: True
Type: GetPetSet
Last Transition Time: 2024-10-25T09:22:10Z
Message: is pet set deleted; ConditionStatus:True
Observed Generation: 1
Status: True
Type: IsPetSetDeleted
Last Transition Time: 2024-10-25T09:22:30Z
Message: successfully deleted the petSets with orphan propagation policy
Observed Generation: 1
Reason: OrphanPetSetPods
Status: True
Type: OrphanPetSetPods
Last Transition Time: 2024-10-25T09:22:35Z
Message: get pod; ConditionStatus:True
Observed Generation: 1
Status: True
Type: GetPod
Last Transition Time: 2024-10-25T09:22:35Z
Message: is ops req patched; ConditionStatus:True
Observed Generation: 1
Status: True
Type: IsOpsReqPatched
Last Transition Time: 2024-10-25T09:22:35Z
Message: create pod; ConditionStatus:True
Observed Generation: 1
Status: True
Type: CreatePod
Last Transition Time: 2024-10-25T09:22:40Z
Message: get pvc; ConditionStatus:True
Observed Generation: 1
Status: True
Type: GetPvc
Last Transition Time: 2024-10-25T09:22:40Z
Message: is pvc patched; ConditionStatus:True
Observed Generation: 1
Status: True
Type: IsPvcPatched
Last Transition Time: 2024-10-25T09:23:50Z
Message: compare storage; ConditionStatus:True
Observed Generation: 1
Status: True
Type: CompareStorage
Last Transition Time: 2024-10-25T09:23:00Z
Message: create; ConditionStatus:True
Observed Generation: 1
Status: True
Type: Create
Last Transition Time: 2024-10-25T09:23:08Z
Message: is druid running; ConditionStatus:False
Observed Generation: 1
Status: False
Type: IsDruidRunning
Last Transition Time: 2024-10-25T09:23:20Z
Message: successfully updated middleManagers node PVC sizes
Observed Generation: 1
Reason: UpdateMiddleManagersNodePVCs
Status: True
Type: UpdateMiddleManagersNodePVCs
Last Transition Time: 2024-10-25T09:24:15Z
Message: successfully updated historicals node PVC sizes
Observed Generation: 1
Reason: UpdateHistoricalsNodePVCs
Status: True
Type: UpdateHistoricalsNodePVCs
Last Transition Time: 2024-10-25T09:24:30Z
Message: successfully reconciled the Druid resources
Observed Generation: 1
Reason: UpdatePetSets
Status: True
Type: UpdatePetSets
Last Transition Time: 2024-10-25T09:24:35Z
Message: PetSet is recreated
Observed Generation: 1
Reason: ReadyPetSets
Status: True
Type: ReadyPetSets
Last Transition Time: 2024-10-25T09:24:35Z
Message: Successfully completed volumeExpansion for Druid
Observed Generation: 1
Reason: Successful
Status: True
Type: Successful
Observed Generation: 1
Phase: Successful
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 10m KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/dr-volume-exp
Normal Starting 10m KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-cluster
Normal Successful 10m KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-cluster for DruidOpsRequest: dr-volume-exp
Warning get pet set; ConditionStatus:True 10m KubeDB Ops-manager Operator get pet set; ConditionStatus:True
Warning is pet set deleted; ConditionStatus:True 10m KubeDB Ops-manager Operator is pet set deleted; ConditionStatus:True
Warning get pet set; ConditionStatus:True 10m KubeDB Ops-manager Operator get pet set; ConditionStatus:True
Warning get pet set; ConditionStatus:True 10m KubeDB Ops-manager Operator get pet set; ConditionStatus:True
Warning is pet set deleted; ConditionStatus:True 10m KubeDB Ops-manager Operator is pet set deleted; ConditionStatus:True
Warning get pet set; ConditionStatus:True 10m KubeDB Ops-manager Operator get pet set; ConditionStatus:True
Normal OrphanPetSetPods 9m59s KubeDB Ops-manager Operator successfully deleted the petSets with orphan propagation policy
Warning get pod; ConditionStatus:True 9m54s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning is ops req patched; ConditionStatus:True 9m54s KubeDB Ops-manager Operator is ops req patched; ConditionStatus:True
Warning create pod; ConditionStatus:True 9m54s KubeDB Ops-manager Operator create pod; ConditionStatus:True
Warning get pod; ConditionStatus:True 9m49s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning get pvc; ConditionStatus:True 9m49s KubeDB Ops-manager Operator get pvc; ConditionStatus:True
Warning is pvc patched; ConditionStatus:True 9m49s KubeDB Ops-manager Operator is pvc patched; ConditionStatus:True
Warning compare storage; ConditionStatus:False 9m49s KubeDB Ops-manager Operator compare storage; ConditionStatus:False
Warning get pod; ConditionStatus:True 9m44s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning get pvc; ConditionStatus:True 9m44s KubeDB Ops-manager Operator get pvc; ConditionStatus:True
Warning get pod; ConditionStatus:True 9m39s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning get pvc; ConditionStatus:True 9m39s KubeDB Ops-manager Operator get pvc; ConditionStatus:True
Warning get pod; ConditionStatus:True 9m34s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning get pvc; ConditionStatus:True 9m34s KubeDB Ops-manager Operator get pvc; ConditionStatus:True
Warning get pod; ConditionStatus:True 9m29s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning get pvc; ConditionStatus:True 9m29s KubeDB Ops-manager Operator get pvc; ConditionStatus:True
Warning compare storage; ConditionStatus:True 9m29s KubeDB Ops-manager Operator compare storage; ConditionStatus:True
Warning create; ConditionStatus:True 9m29s KubeDB Ops-manager Operator create; ConditionStatus:True
Warning is ops req patched; ConditionStatus:True 9m29s KubeDB Ops-manager Operator is ops req patched; ConditionStatus:True
Warning get pod; ConditionStatus:True 9m24s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning is druid running; ConditionStatus:False 9m21s KubeDB Ops-manager Operator is druid running; ConditionStatus:False
Warning get pod; ConditionStatus:True 9m19s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning get pod; ConditionStatus:True 9m14s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Normal UpdateMiddleManagersNodePVCs 9m9s KubeDB Ops-manager Operator successfully updated middleManagers node PVC sizes
Warning get pod; ConditionStatus:True 9m4s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning is ops req patched; ConditionStatus:True 9m4s KubeDB Ops-manager Operator is ops req patched; ConditionStatus:True
Warning create pod; ConditionStatus:True 9m4s KubeDB Ops-manager Operator create pod; ConditionStatus:True
Warning get pod; ConditionStatus:True 8m59s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning get pvc; ConditionStatus:True 8m59s KubeDB Ops-manager Operator get pvc; ConditionStatus:True
Warning is pvc patched; ConditionStatus:True 8m59s KubeDB Ops-manager Operator is pvc patched; ConditionStatus:True
Warning compare storage; ConditionStatus:False 8m59s KubeDB Ops-manager Operator compare storage; ConditionStatus:False
Warning get pod; ConditionStatus:True 8m54s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning get pvc; ConditionStatus:True 8m54s KubeDB Ops-manager Operator get pvc; ConditionStatus:True
Warning get pod; ConditionStatus:True 8m49s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning get pvc; ConditionStatus:True 8m49s KubeDB Ops-manager Operator get pvc; ConditionStatus:True
Warning get pod; ConditionStatus:True 8m44s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning get pvc; ConditionStatus:True 8m44s KubeDB Ops-manager Operator get pvc; ConditionStatus:True
Warning get pod; ConditionStatus:True 8m39s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning get pvc; ConditionStatus:True 8m39s KubeDB Ops-manager Operator get pvc; ConditionStatus:True
Warning compare storage; ConditionStatus:True 8m39s KubeDB Ops-manager Operator compare storage; ConditionStatus:True
Warning create; ConditionStatus:True 8m39s KubeDB Ops-manager Operator create; ConditionStatus:True
Warning is ops req patched; ConditionStatus:True 8m39s KubeDB Ops-manager Operator is ops req patched; ConditionStatus:True
Warning get pod; ConditionStatus:True 8m34s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning is druid running; ConditionStatus:False 8m31s KubeDB Ops-manager Operator is druid running; ConditionStatus:False
Warning get pod; ConditionStatus:True 8m29s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning get pod; ConditionStatus:True 8m24s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Warning get pod; ConditionStatus:True 8m19s KubeDB Ops-manager Operator get pod; ConditionStatus:True
Normal UpdateHistoricalsNodePVCs 8m14s KubeDB Ops-manager Operator successfully updated historicals node PVC sizes
Normal UpdatePetSets 7m59s KubeDB Ops-manager Operator successfully reconciled the Druid resources
Warning get pet set; ConditionStatus:True 7m54s KubeDB Ops-manager Operator get pet set; ConditionStatus:True
Warning get pet set; ConditionStatus:True 7m54s KubeDB Ops-manager Operator get pet set; ConditionStatus:True
Normal ReadyPetSets 7m54s KubeDB Ops-manager Operator PetSet is recreated
Normal Starting 7m54s KubeDB Ops-manager Operator Resuming Druid database: demo/druid-cluster
Normal Successful 7m54s KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-cluster for DruidOpsRequest: dr-volume-exp
Now, we are going to verify from the Petset
, and the Persistent Volumes
whether the volume of the database has expanded to meet the desired state, Let’s check,
$ kubectl get petset -n demo druid-cluster-historicals -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage'
"3Gi"
$ kubectl get petset -n demo druid-cluster-middleManagers -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage'
"2Gi"
$ kubectl get pv -n demo
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-0bf49077-1c7a-4943-bb17-1dffd1626dcd 2Gi RWO Delete Bound demo/druid-cluster-segment-cache-druid-cluster-historicals-0 longhorn <unset> 23m
pvc-59ed4914-53b3-4f18-a6aa-7699c2b738e2 2Gi RWO Delete Bound demo/druid-cluster-base-task-dir-druid-cluster-middlemanagers-0 longhorn <unset> 23m
The above output verifies that we have successfully expanded the volume of the Druid.
Cleaning Up
To clean up the Kubernetes resources created by this tutorial, run:
kubectl delete druidopsrequest -n demo dr-volume-exp
kubectl delete dr -n demo druid-cluster
kubectl delete ns demo
Next Steps
- Detail concepts of Druid object.
- Different Druid topology clustering modes here.
- Monitor your Druid database with KubeDB using out-of-the-box Prometheus operator.
- Want to hack on KubeDB? Check our contribution guidelines.