You are looking at the documentation of a prior release. To read the documentation of the latest release, please
visit here.
New to KubeDB? Please start here.
Storage Autoscaling of a PerconaXtraDB Cluster
This guide will show you how to use KubeDB
to autoscale the storage of a PerconaXtraDB Replicaset database.
Before You Begin
At first, you need to have a Kubernetes cluster, and the
kubectl
command-line tool must be configured to communicate with your cluster.Install
KubeDB
Community, Enterprise and Autoscaler operator in your cluster following the steps here.Install
Metrics Server
from hereInstall Prometheus from here
You must have a
StorageClass
that supports volume expansion.You should be familiar with the following
KubeDB
concepts:
To keep everything isolated, we are going to use a separate namespace called demo
throughout this tutorial.
$ kubectl create ns demo
namespace/demo created
Storage Autoscaling of Cluster Database
At first verify that your cluster has a storage class, that supports volume expansion. Let’s check,
$ kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
standard (default) rancher.io/local-path Delete WaitForFirstConsumer false 79m
topolvm-provisioner topolvm.cybozu.com Delete WaitForFirstConsumer true 78m
We can see from the output the topolvm-provisioner
storage class has ALLOWVOLUMEEXPANSION
field as true. So, this storage class supports volume expansion. We can use it. You can install topolvm from here
Now, we are going to deploy a PerconaXtraDB
replicaset using a supported version by KubeDB
operator. Then we are going to apply PerconaXtraDBAutoscaler
to set up autoscaling.
Deploy PerconaXtraDB Cluster
In this section, we are going to deploy a PerconaXtraDB replicaset database with version 8.0.26
. Then, in the next section we will set up autoscaling for this database using PerconaXtraDBAutoscaler
CRD. Below is the YAML of the PerconaXtraDB
CR that we are going to create,
If you want to autoscale PerconaXtraDB
Standalone
, Just remove thespec.Replicas
from the below yaml and rest of the steps are same.
apiVersion: kubedb.com/v1
kind: PerconaXtraDB
metadata:
name: sample-pxc
namespace: demo
spec:
version: "8.0.26"
replicas: 3
storageType: Durable
storage:
storageClassName: "topolvm-provisioner"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
deletionPolicy: WipeOut
Let’s create the PerconaXtraDB
CRO we have shown above,
$ kubectl create -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/guides/percona-xtradb/autoscaler/storage/cluster/examples/sample-pxc.yaml
perconaxtradb.kubedb.com/sample-pxc created
Now, wait until sample-pxc
has status Ready
. i.e,
$ kubectl get perconaxtradb -n demo
NAME VERSION STATUS AGE
sample-pxc 8.0.26 Ready 3m46s
Let’s check volume size from petset, and from the persistent volume,
$ kubectl get sts -n demo sample-pxc -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage'
"1Gi"
$ kubectl get pv -n demo
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-43266d76-f280-4cca-bd78-d13660a84db9 1Gi RWO Delete Bound demo/data-sample-pxc-2 topolvm-provisioner 57s
pvc-4a509b05-774b-42d9-b36d-599c9056af37 1Gi RWO Delete Bound demo/data-sample-pxc-0 topolvm-provisioner 58s
pvc-c27eee12-cd86-4410-b39e-b1dd735fc14d 1Gi RWO Delete Bound demo/data-sample-pxc-1 topolvm-provisioner 57s
You can see the petset has 1GB storage, and the capacity of all the persistent volume is also 1GB.
We are now ready to apply the PerconaXtraDBAutoscaler
CRO to set up storage autoscaling for this database.
Storage Autoscaling
Here, we are going to set up storage autoscaling using a PerconaXtraDBAutoscaler Object.
Create PerconaXtraDBAutoscaler Object
In order to set up vertical autoscaling for this replicaset database, we have to create a PerconaXtraDBAutoscaler
CRO with our desired configuration. Below is the YAML of the PerconaXtraDBAutoscaler
object that we are going to create,
apiVersion: autoscaling.kubedb.com/v1alpha1
kind: PerconaXtraDBAutoscaler
metadata:
name: px-as-st
namespace: demo
spec:
databaseRef:
name: sample-pxc
storage:
perconaxtradb:
trigger: "On"
usageThreshold: 20
scalingThreshold: 20
expansionMode: "Online"
Here,
spec.databaseRef.name
specifies that we are performing vertical scaling operation onsample-pxc
database.spec.storage.perconaxtradb.trigger
specifies that storage autoscaling is enabled for this database.spec.storage.perconaxtradb.usageThreshold
specifies storage usage threshold, if storage usage exceeds20%
then storage autoscaling will be triggered.spec.storage.perconaxtradb.scalingThreshold
specifies the scaling threshold. Storage will be scaled to20%
of the current amount.spec.storage.perconaxtradb.expansionMode
specifies the expansion mode of volume expansionPerconaXtraDBOpsRequest
created byPerconaXtraDBAutoscaler
. topolvm-provisioner supports online volume expansion so hereexpansionMode
is set as “Online”.
Let’s create the PerconaXtraDBAutoscaler
CR we have shown above,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/guides/percona-xtradb/autoscaler/storage/cluster/examples/pxas-storage.yaml
perconaxtradbautoscaler.autoscaling.kubedb.com/px-as-st created
Storage Autoscaling is set up successfully
Let’s check that the perconaxtradbautoscaler
resource is created successfully,
$ kubectl get perconaxtradbautoscaler -n demo
NAME AGE
px-as-st 33s
$ kubectl describe perconaxtradbautoscaler px-as-st -n demo
Name: px-as-st
Namespace: demo
Labels: <none>
Annotations: API Version: autoscaling.kubedb.com/v1alpha1
Kind: PerconaXtraDBAutoscaler
Metadata:
Creation Timestamp: 2022-01-14T06:08:02Z
Generation: 1
Managed Fields:
...
Resource Version: 24009
UID: 4f45a3b3-fc72-4d04-b52c-a770944311f6
Spec:
Database Ref:
Name: sample-pxc
Storage:
Mariadb:
Scaling Threshold: 20
Trigger: On
Usage Threshold: 20
Events: <none>
So, the perconaxtradbautoscaler
resource is created successfully.
Now, for this demo, we are going to manually fill up the persistent volume to exceed the usageThreshold
using dd
command to see if storage autoscaling is working or not.
Let’s exec into the database pod and fill the database volume(var/lib/mysql
) using the following commands:
$ kubectl exec -it -n demo sample-pxc-0 -- bash
root@sample-pxc-0:/ df -h /var/lib/mysql
Filesystem Size Used Avail Use% Mounted on
/dev/topolvm/57cd4330-784f-42c1-bf8e-e743241df164 1014M 357M 658M 36% /var/lib/mysql
root@sample-pxc-0:/ dd if=/dev/zero of=/var/lib/mysql/file.img bs=500M count=1
1+0 records in
1+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 0.340877 s, 1.5 GB/s
root@sample-pxc-0:/ df -h /var/lib/mysql
Filesystem Size Used Avail Use% Mounted on
/dev/topolvm/57cd4330-784f-42c1-bf8e-e743241df164 1014M 857M 158M 85% /var/lib/mysql
So, from the above output we can see that the storage usage is 83%, which exceeded the usageThreshold
20%.
Let’s watch the perconaxtradbopsrequest
in the demo namespace to see if any perconaxtradbopsrequest
object is created. After some time you’ll see that a perconaxtradbopsrequest
of type VolumeExpansion
will be created based on the scalingThreshold
.
$ kubectl get perconaxtradbopsrequest -n demo
NAME TYPE STATUS AGE
mops-sample-pxc-xojkua VolumeExpansion Progressing 15s
Let’s wait for the ops request to become successful.
$ kubectl get perconaxtradbopsrequest -n demo
NAME TYPE STATUS AGE
mops-sample-pxc-xojkua VolumeExpansion Successful 97s
We can see from the above output that the PerconaXtraDBOpsRequest
has succeeded. If we describe the PerconaXtraDBOpsRequest
we will get an overview of the steps that were followed to expand the volume of the database.
$ kubectl describe perconaxtradbopsrequest -n demo mops-sample-pxc-xojkua
Name: mops-sample-pxc-xojkua
Namespace: demo
Labels: app.kubernetes.io/component=database
app.kubernetes.io/instance=sample-pxc
app.kubernetes.io/managed-by=kubedb.com
app.kubernetes.io/name=perconaxtradbs.kubedb.com
Annotations: <none>
API Version: ops.kubedb.com/v1alpha1
Kind: PerconaXtraDBOpsRequest
Metadata:
Creation Timestamp: 2022-01-14T06:13:10Z
Generation: 1
Managed Fields: ...
Owner References:
API Version: autoscaling.kubedb.com/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: PerconaXtraDBAutoscaler
Name: px-as-st
UID: 4f45a3b3-fc72-4d04-b52c-a770944311f6
Resource Version: 25557
UID: 90763a49-a03f-407c-a233-fb20c4ab57d7
Spec:
Database Ref:
Name: sample-pxc
Type: VolumeExpansion
Volume Expansion:
Mariadb: 1594884096
Status:
Conditions:
Last Transition Time: 2022-01-14T06:13:10Z
Message: Controller has started to Progress the PerconaXtraDBOpsRequest: demo/mops-sample-pxc-xojkua
Observed Generation: 1
Reason: OpsRequestProgressingStarted
Status: True
Type: Progressing
Last Transition Time: 2022-01-14T06:14:25Z
Message: Volume Expansion performed successfully in PerconaXtraDB pod for PerconaXtraDBOpsRequest: demo/mops-sample-pxc-xojkua
Observed Generation: 1
Reason: SuccessfullyVolumeExpanded
Status: True
Type: VolumeExpansion
Last Transition Time: 2022-01-14T06:14:25Z
Message: Controller has successfully expand the volume of PerconaXtraDB demo/mops-sample-pxc-xojkua
Observed Generation: 1
Reason: OpsRequestProcessedSuccessfully
Status: True
Type: Successful
Observed Generation: 3
Phase: Successful
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 2m58s KubeDB Enterprise Operator Start processing for PerconaXtraDBOpsRequest: demo/mops-sample-pxc-xojkua
Normal Starting 2m58s KubeDB Enterprise Operator Pausing PerconaXtraDB databse: demo/sample-pxc
Normal Successful 2m58s KubeDB Enterprise Operator Successfully paused PerconaXtraDB database: demo/sample-pxc for PerconaXtraDBOpsRequest: mops-sample-pxc-xojkua
Normal Successful 103s KubeDB Enterprise Operator Volume Expansion performed successfully in PerconaXtraDB pod for PerconaXtraDBOpsRequest: demo/mops-sample-pxc-xojkua
Normal Starting 103s KubeDB Enterprise Operator Updating PerconaXtraDB storage
Normal Successful 103s KubeDB Enterprise Operator Successfully Updated PerconaXtraDB storage
Normal Starting 103s KubeDB Enterprise Operator Resuming PerconaXtraDB database: demo/sample-pxc
Normal Successful 103s KubeDB Enterprise Operator Successfully resumed PerconaXtraDB database: demo/sample-pxc
Normal Successful 103s KubeDB Enterprise Operator Controller has Successfully expand the volume of PerconaXtraDB: demo/sample-pxc
Now, we are going to verify from the Petset
, and the Persistent Volume
whether the volume of the replicaset database has expanded to meet the desired state, Let’s check,
$ kubectl get sts -n demo sample-pxc -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage'
"1594884096"
$ kubectl get pv -n demo
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-43266d76-f280-4cca-bd78-d13660a84db9 2Gi RWO Delete Bound demo/data-sample-pxc-2 topolvm-provisioner 23m
pvc-4a509b05-774b-42d9-b36d-599c9056af37 2Gi RWO Delete Bound demo/data-sample-pxc-0 topolvm-provisioner 24m
pvc-c27eee12-cd86-4410-b39e-b1dd735fc14d 2Gi RWO Delete Bound demo/data-sample-pxc-1 topolvm-provisioner 23m
The above output verifies that we have successfully autoscaled the volume of the PerconaXtraDB replicaset database.
Cleaning Up
To clean up the Kubernetes resources created by this tutorial, run:
kubectl delete perconaxtradb -n demo sample-pxc
kubectl delete perconaxtradbautoscaler -n demo px-as-st
kubectl delete ns demo