You are looking at the documentation of a prior release. To read the documentation of the latest release, please
visit here.
New to KubeDB? Please start here.
Autoscaling the Compute Resource of an Elasticsearch Combined Cluster
This guide will show you how to use KubeDB to autoscale compute resources i.e. cpu and memory of an Elasticsearch combined cluster.
Before You Begin
At first, you need to have a Kubernetes cluster, and the
kubectlcommand-line tool must be configured to communicate with your cluster.Install
KubeDBCommunity, Enterprise and Autoscaler operator in your cluster following the steps here.Install
Metrics Serverfrom hereYou should be familiar with the following
KubeDBconcepts:
To keep everything isolated, we are going to use a separate namespace called demo throughout this tutorial.
$ kubectl create ns demo
namespace/demo created
Note: YAML files used in this tutorial are stored in this directory of kubedb/docs repository.
Autoscaling of a Combined Cluster
Here, we are going to deploy an Elasticsearch in combined cluster mode using a supported version by KubeDB operator. Then we are going to apply ElasticsearchAutoscaler to set up autoscaling.
Deploy Elasticsearch standalone
In this section, we are going to deploy an Elasticsearch combined cluster with ElasticsearchVersion xpack-8.11.1. Then, in the next section, we will set up autoscaling for this database using ElasticsearchAutoscaler CRD. Below is the YAML of the Elasticsearch CR that we are going to create,
apiVersion: kubedb.com/v1
kind: Elasticsearch
metadata:
name: es-combined
namespace: demo
spec:
enableSSL: true
version: xpack-8.2.3
storageType: Durable
replicas: 3
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
podTemplate:
spec:
containers:
- name: elasticsearch
resources:
requests:
cpu: "500m"
limits:
cpu: "500m"
memory: "1.2Gi"
deletionPolicy: WipeOut
Let’s create the Elasticsearch CRO we have shown above,
$ kubectl create -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/guides/elasticsearch/autoscaler/compute/combined/yamls/es-combined.yaml
elasticsearch.kubedb.com/es-combined created
Now, wait until es-combined has status Ready. i.e,
$ kubectl get elasticsearch -n demo -w
NAME VERSION STATUS AGE
es-combined xpack-8.2.3 Provisioning 4s
es-combined xpack-8.2.3 Provisioning 7s
....
....
es-combined xpack-8.2.3 Ready 60s
Let’s check the Pod containers resources,
$ kubectl get pod -n demo es-combined-0 -o json | jq '.spec.containers[].resources'
{
"limits": {
"cpu": "500m",
"memory": "1288490188800m"
},
"requests": {
"cpu": "500m",
"memory": "1288490188800m"
}
}
Let’s check the Elasticsearch resources,
$ kubectl get elasticsearch -n demo es-combined -o json | jq '.spec.podTemplate.spec.containers[] | select(.name == "elasticsearch") | .resources'
{
"limits": {
"cpu": "500m",
"memory": "1288490188800m"
},
"requests": {
"cpu": "500m",
"memory": "1288490188800m"
}
}
You can see from the above outputs that the resources are the same as the ones we have assigned while deploying the Elasticsearch.
We are now ready to apply the ElasticsearchAutoscaler CRO to set up autoscaling for this database.
Compute Resource Autoscaling
Here, we are going to set up compute (ie. cpu and memory) autoscaling using an ElasticsearchAutoscaler Object.
Create ElasticsearchAutoscaler Object
To set up compute resource autoscaling for this combined cluster, we have to create a ElasticsearchAutoscaler CRO with our desired configuration. Below is the YAML of the ElasticsearchAutoscaler object that we are going to create,
apiVersion: autoscaling.kubedb.com/v1alpha1
kind: ElasticsearchAutoscaler
metadata:
name: es-combined-as
namespace: demo
spec:
databaseRef:
name: es-combined
compute:
node:
trigger: "On"
podLifeTimeThreshold: 5m
resourceDiffPercentage: 5
minAllowed:
cpu: 1
memory: "2.1Gi"
maxAllowed:
cpu: 2
memory: 3Gi
controlledResources: ["cpu", "memory"]
containerControlledValues: "RequestsAndLimits"
Here,
spec.databaseRef.namespecifies that we are performing compute resource autoscaling ones-combineddatabase.spec.compute.node.triggerspecifies that compute resource autoscaling is enabled for this cluster.spec.compute.node.podLifeTimeThresholdspecifies the minimum lifetime for at least one of the pod to initiate a vertical scaling.spec.compute.node.minAllowedspecifies the minimum allowed resources for the Elasticsearch node.spec.compute.node.maxAllowedspecifies the maximum allowed resources for the Elasticsearch node.spec.compute.node.controlledResourcesspecifies the resources that are controlled by the autoscaler.spec.compute.node.resourceDiffPercentagespecifies the minimum resource difference in percentage. The default is 10%. If the difference between current & recommended resource is less than ResourceDiffPercentage, Autoscaler Operator will ignore the updating.spec.compute.node.containerControlledValuesspecifies which resource values should be controlled. The default is “RequestsAndLimits”.
Let’s create the ElasticsearchAutoscaler CR we have shown above,
$ kubectl apply -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/guides/elasticsearch/autoscaler/compute/combined/yamls/es-auto-scaler.yaml
elasticsearchautoscaler.autoscaling.kubedb.com/es-combined-as created
Verify Autoscaling is set up successfully
Let’s check that the elasticsearchautoscaler resource is created successfully,
$kubectl get elasticsearchautoscaler -n demo
NAME AGE
es-combined-as 14s
$ kubectl describe elasticsearchautoscaler -n demo es-combined-as
Name: es-combined-as
Namespace: demo
Labels: <none>
Annotations: <none>
API Version: autoscaling.kubedb.com/v1alpha1
Kind: ElasticsearchAutoscaler
Metadata:
Creation Timestamp: 2022-12-29T10:54:00Z
Generation: 1
Managed Fields:
API Version: autoscaling.kubedb.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:compute:
.:
f:node:
.:
f:containerControlledValues:
f:controlledResources:
f:maxAllowed:
.:
f:cpu:
f:memory:
f:minAllowed:
.:
f:cpu:
f:memory:
f:podLifeTimeThreshold:
f:resourceDiffPercentage:
f:trigger:
f:databaseRef:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-12-29T10:54:00Z
API Version: autoscaling.kubedb.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:checkpoints:
f:conditions:
f:vpas:
Manager: kubedb-autoscaler
Operation: Update
Subresource: status
Time: 2022-12-29T10:54:27Z
Resource Version: 12469
UID: 35640903-7aaf-46c6-9bc4-bd1771313e30
Spec:
Compute:
Node:
Container Controlled Values: RequestsAndLimits
Controlled Resources:
cpu
memory
Max Allowed:
Cpu: 2
Memory: 3Gi
Min Allowed:
Cpu: 1
Memory: 2254857830400m
Pod Life Time Threshold: 5m0s
Resource Diff Percentage: 5
Trigger: On
Database Ref:
Name: es-combined
Ops Request Options:
Apply: IfReady
Status:
Checkpoints:
Cpu Histogram:
Bucket Weights:
Index: 0
Weight: 2849
Index: 1
Weight: 10000
Index: 2
Weight: 2856
Index: 3
Weight: 714
Index: 5
Weight: 714
Index: 6
Weight: 713
Index: 7
Weight: 714
Index: 12
Weight: 713
Index: 21
Weight: 713
Index: 25
Weight: 2138
Reference Timestamp: 2022-12-29T00:00:00Z
Total Weight: 4.257959878725071
First Sample Start: 2022-12-29T10:54:03Z
Last Sample Start: 2022-12-29T11:04:18Z
Last Update Time: 2022-12-29T11:04:26Z
Memory Histogram:
Reference Timestamp: 2022-12-30T00:00:00Z
Ref:
Container Name: elasticsearch
Vpa Object Name: es-combined
Total Samples Count: 31
Version: v3
Conditions:
Last Transition Time: 2022-12-29T10:54:27Z
Message: Successfully created elasticsearchOpsRequest demo/esops-es-combined-ujb5hy
Observed Generation: 1
Reason: CreateOpsRequest
Status: True
Type: CreateOpsRequest
Vpas:
Conditions:
Last Transition Time: 2022-12-29T10:54:26Z
Status: True
Type: RecommendationProvided
Recommendation:
Container Recommendations:
Container Name: elasticsearch
Lower Bound:
Cpu: 1
Memory: 2254857830400m
Target:
Cpu: 1
Memory: 2254857830400m
Uncapped Target:
Cpu: 442m
Memory: 1555165137
Upper Bound:
Cpu: 2
Memory: 3Gi
Vpa Name: es-combined
Events: <none>
So, the elasticsearchautoscaler resource is created successfully.
you can see in the Status.VPAs.Recommendation section, that recommendation has been generated for our database. Our autoscaler operator continuously watches the recommendation generated and creates an elasticsearchopsrequest based on the recommendations, if the database pods are needed to scaled up or down.
Let’s watch the elasticsearchopsrequest in the demo namespace to see if any elasticsearchopsrequest object is created. After some time you’ll see that an elasticsearchopsrequest will be created based on the recommendation.
$ kubectl get elasticsearchopsrequest -n demo
NAME TYPE STATUS AGE
esops-es-combined-ujb5hy VerticalScaling Progessing 1m
Let’s wait for the opsRequest to become successful.
$ kubectl get elasticsearchopsrequest -n demo
NAME TYPE STATUS AGE
esops-es-combined-ujb5hy VerticalScaling Successful 1m
We can see from the above output that the ElasticsearchOpsRequest has succeeded. If we describe the ElasticsearchOpsRequest we will get an overview of the steps that were followed to scale the database.
$ kubectl describe elasticsearchopsrequest -n demo esops-es-combined-ujb5hy
Name: esops-es-combined-ujb5hy
Namespace: demo
Labels: <none>
Annotations: <none>
API Version: ops.kubedb.com/v1alpha1
Kind: ElasticsearchOpsRequest
Metadata:
Creation Timestamp: 2022-12-29T10:54:27Z
Generation: 1
Managed Fields:
API Version: ops.kubedb.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:ownerReferences:
.:
k:{"uid":"35640903-7aaf-46c6-9bc4-bd1771313e30"}:
f:spec:
.:
f:apply:
f:databaseRef:
f:type:
f:verticalScaling:
.:
f:node:
.:
f:limits:
.:
f:cpu:
f:memory:
f:requests:
.:
f:cpu:
f:memory:
Manager: kubedb-autoscaler
Operation: Update
Time: 2022-12-29T10:54:27Z
API Version: ops.kubedb.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
f:observedGeneration:
f:phase:
Manager: kubedb-ops-manager
Operation: Update
Subresource: status
Time: 2022-12-29T10:54:27Z
Owner References:
API Version: autoscaling.kubedb.com/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: ElasticsearchAutoscaler
Name: es-combined-as
UID: 35640903-7aaf-46c6-9bc4-bd1771313e30
Resource Version: 11992
UID: 4aa5295f-0702-45ac-9ae8-3cb496b0e740
Spec:
Apply: IfReady
Database Ref:
Name: es-combined
Type: VerticalScaling
Vertical Scaling:
Node:
Limits:
Cpu: 1
Memory: 2254857830400m
Requests:
Cpu: 1
Memory: 2254857830400m
Status:
Conditions:
Last Transition Time: 2022-12-29T10:54:27Z
Message: Elasticsearch ops request is vertically scaling the nodes
Observed Generation: 1
Reason: VerticalScaling
Status: True
Type: VerticalScaling
Last Transition Time: 2022-12-29T10:54:39Z
Message: successfully reconciled the Elasticsearch resources
Observed Generation: 1
Reason: Reconciled
Status: True
Type: Reconciled
Last Transition Time: 2022-12-29T10:58:39Z
Message: Successfully restarted all nodes
Observed Generation: 1
Reason: RestartNodes
Status: True
Type: RestartNodes
Last Transition Time: 2022-12-29T10:58:44Z
Message: successfully updated Elasticsearch CR
Observed Generation: 1
Reason: UpdateElasticsearchCR
Status: True
Type: UpdateElasticsearchCR
Last Transition Time: 2022-12-29T10:58:45Z
Message: Successfully completed the modification process.
Observed Generation: 1
Reason: Successful
Status: True
Type: Successful
Observed Generation: 1
Phase: Successful
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal PauseDatabase 8m25s KubeDB Ops-manager Operator Pausing Elasticsearch demo/es-combined
Normal Reconciled 8m13s KubeDB Ops-manager Operator successfully reconciled the Elasticsearch resources
Normal RestartNodes 4m13s KubeDB Ops-manager Operator Successfully restarted all nodes
Normal UpdateElasticsearchCR 4m7s KubeDB Ops-manager Operator successfully updated Elasticsearch CR
Normal ResumeDatabase 4m7s KubeDB Ops-manager Operator Resuming Elasticsearch demo/es-combined
Normal Successful 4m7s KubeDB Ops-manager Operator Successfully Updated Database
Now, we are going to verify from the Pod, and the Elasticsearch YAML whether the resources of the standalone database has updated to meet up the desired state, Let’s check,
$ kubectl get pod -n demo es-combined-0 -o json | jq '.spec.containers[].resources'
{
"limits": {
"cpu": "500m",
"memory": "1288490188800m"
},
"requests": {
"cpu": "500m",
"memory": "1288490188800m"
}
}
$ kubectl get elasticsearch -n demo es-combined -o json | jq '.spec.podTemplate.spec.containers[] | select(.name == "elasticsearch") | .resources'
{
"limits": {
"cpu": "1",
"memory": "2254857830400m"
},
"requests": {
"cpu": "1",
"memory": "2254857830400m"
}
}
The above output verifies that we have successfully auto-scaled the resources of the Elasticsearch standalone database.
Cleaning Up
To clean up the Kubernetes resources created by this tutorial, run:
$ kubectl delete es -n demo es-combined
$ kubectl delete elasticsearchautoscaler -n demo es-combined-as
$ kubectl delete ns demo