Deploy Elasticsearch via Kubernetes Elasticsearch Operator
In today’s environment of containerized applications and orchestration, effectively deploying and maintaining databases such as Elasticsearch demands an flexible and effective strategy. The Kubernetes Elasticsearch Operator simplifies the task of setting up, maintaining, and expanding Elasticsearch databases within a Kubernetes framework. This document explores the basics of Elasticsearch database installation through the Elasticsearch Kubernetes Operator, analyzing its advantages, characteristics, and stepwise instructions. Through this solution, you can attain increased adaptability and automation when it comes to provisioning and managing your Elasticsearch databases, while also following containerized infrastructure best practices.
Kubernetes is a revolutionary open-source platform that simplifies the whole process of containerized applications. With Kubernetes, you can acquire the ability to effortlessly launch, expand, and oversee applications, whether they operate on a single machine or span across a multi-cloud environment.
Kubernetes simplifies the management of numerous containers by automating essential tasks like load balancing, real-time scaling, and ensuring application resilience through automatic recovery. When it’s time to release a new version of your application, Kubernetes handles updates, minimizing downtime and reducing the potential for errors.
With a simple declarative configuration, you can specify your desired application behavior, and Kubernetes ensures it follows those specifications. This allows you to concentrate on developing exceptional applications, while Kubernetes guarantees their reliable and efficient operation. Now, you can simplify the provisioning and troubleshooting process and empowering you to confidently tackle the complexity of application deployment.
Why Elasticsearch in Kubernetes
Elasticsearch is an open-source, highly scalable search engine designed to handle massive amounts of data. It has many features, including full-text search, structured search, analytics, logging, and can be applied to a wide range of applications and use cases. Elasticsearch is an excellent choice for enterprises that need to manage and search through massive amounts of information in real-time since it provides fast and accurate search results.
Elasticsearch employs a distributed architecture that enables horizontal scaling across multiple nodes, ensuring it can effectively manage extensive data loads while ensuring continuous accessibility. In addition to its distributed design, Elasticsearch supports a flexible data format, enabling you to store and index various data types, including text, numerical data, and geospatial information, both structured and unstructured.
Integrating Elasticsearch within a Kubernetes environment offers a powerful combination that brings a host of advantages. It allows for the seamless management of Elasticsearch clusters at scale, ensuring optimal resource utilization and high availability, all within the robust orchestration framework of Kubernetes. Kubernetes simplifies the deployment and scaling of Elasticsearch instances, making it easier to adapt to evolving data demands. Additionally, it provides a unified platform for handling both application and data infrastructure, streamlining operations and reducing complexity. This integration enhances the overall efficiency and resilience of Elasticsearch deployments, facilitating real-time data processing and search capabilities within Kubernetes clusters, making it a formidable solution for modern data-driven applications.
Deploy Elasticsearch on Kubernetes
Pre-requisites
We have to set up the environment to deploy Elasticsearch on Kubernetes using a Kubernetes Elasticsearch operator. A running Kubernetes cluster and a fundamental understanding of Elasticsearch are required to conduct this tutorial. Here, we are going to create our kubernetes cluster using Kind. Additionally, you need to install Helm to your Kubernetes cluster.
In this article, We will use the Kubernetes Elasticsearch operator KubeDB to deploy Elasticsearch on Kubernetes. We must have KubeDB installed in our Kubernetes cluster. To set up KubeDB in our Kubernetes cluster, we require a license. Through the Appscode License Server, we can get a free enterprise license. We must provide our Kubernetes cluster ID to obtain a license. Run the following command below to get the cluster ID.
$ kubectl get ns kube-system -o jsonpath='{.metadata.uid}'
6c08dcb8-8440-4388-849f-1f2b590b731e
The license server will email us with a “license.txt” file attached after we provide the necessary data. Run the following commands listed below to install KubeDB.
$ helm install kubedb oci://ghcr.io/appscode-charts/kubedb \
--version v2023.12.11 \
--namespace kubedb --create-namespace \
--set-file global.license=/path/to/the/license.txt \
--wait --burst-limit=10000 --debug
Verify the installation by the following command,
$ kubectl get pods --all-namespaces -l "app.kubernetes.io/instance=kubedb"
NAMESPACE NAME READY STATUS RESTARTS AGE
kubedb kubedb-kubedb-autoscaler-b99bbb595-kr96z 1/1 Running 0 2m33s
kubedb kubedb-kubedb-dashboard-69b476d689-tglfl 1/1 Running 0 2m33s
kubedb kubedb-kubedb-ops-manager-7c6c7b6c97-9h25q 1/1 Running 0 2m33s
kubedb kubedb-kubedb-provisioner-66f9cb9b99-pfrzb 1/1 Running 0 2m33s
kubedb kubedb-kubedb-schema-manager-7654cd4457-d7s7j 1/1 Running 0 2m33s
kubedb kubedb-kubedb-webhook-server-f89bc674b-g7lcb 1/1 Running 0 2m33s
We can go on to the next stage if every pod status is running.
Create a Namespace
Now we’ll create a new namespace in which we will deploy Elasticsearch. To create a namespace, we can use the following command:
$ kubectl create namespace es-demo
namespace/es-demo created
Deploy Elasticsearch via Kubernetes Elasticsearch operator
We need to create a yaml configuration to deploy Elasticsearch on Kubernetes. And we will apply this yaml below,
apiVersion: kubedb.com/v1alpha2
kind: Elasticsearch
metadata:
name: es-topology-cluster
namespace: es-demo
spec:
enableSSL: true
version: xpack-8.11.1
storageType: Durable
topology:
master:
replicas: 2
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 512Mi
data:
replicas: 2
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 512Mi
ingest:
replicas: 2
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 512Mi
You can see the detailed yaml specifications in the Kubernetes Elasticsearch documentation.
We will save this yaml configuration to es-topology-cluster.yaml. Then create the above Elasticsearch object.
$ kubectl apply -f es-topology-cluster.yaml
elasticsearch.kubedb.com/es-topology-cluster created
If all the above steps are handled correctly and the Elasticsearch is deployed, you will see that the following objects are created:
$ kubectl get all -n es-demo
NAME READY STATUS RESTARTS AGE
pod/es-topology-cluster-data-0 1/1 Running 0 4m
pod/es-topology-cluster-data-1 1/1 Running 0 4m
pod/es-topology-cluster-ingest-0 1/1 Running 0 4m
pod/es-topology-cluster-ingest-1 1/1 Running 0 4m
pod/es-topology-cluster-master-0 1/1 Running 0 4m
pod/es-topology-cluster-master-1 1/1 Running 0 4m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/es-topology-cluster ClusterIP 10.8.11.95 <none> 9200/TCP 4m
service/es-topology-cluster-master ClusterIP None <none> 9300/TCP 4m
service/es-topology-cluster-pods ClusterIP None <none> 9200/TCP 4m
NAME READY AGE
statefulset.apps/es-topology-cluster-data 2/2 4m
statefulset.apps/es-topology-cluster-ingest 2/2 4m
statefulset.apps/es-topology-cluster-master 2/2 4m
NAME TYPE VERSION AGE
appbinding.appcatalog.appscode.com/es-topology-cluster kubedb.com/elasticsearch 8.5.2 4m
NAME VERSION STATUS AGE
elasticsearch.kubedb.com/es-topology-cluster xpack-8.11.1 Ready 4m
We have successfully deployed Elasticsearch to Kubernetes via the Kubernetes Elasticsearch operator. Now, we will connect to the Elasticsearch database to insert some sample data and verify whether our Elasticsearch is usable or not. First, check the database status,
$ kubectl get elasticsearch -n es-demo es-topology-cluster
NAME VERSION STATUS AGE
es-topology-cluster xpack-8.11.1 Ready 4m
Insert sample data to the Elasticsearch database
Now, we will create few indexes in Elasticsearch. The Kubernetes Elasticsearch operator establishes a governing service with the name of the Elasticsearch object itself when Elasticsearch yaml is deployed. Using this service, we will port-forward to the database from our local workstation and establish a connection. After that, we’ll add some data to Elasticsearch.
Port-forward the Service
KubeDB will create few Services to connect with the database. Let’s see the Services created by KubeDB for our Elasticsearch,
$ kubectl get service -n es-demo -l=app.kubernetes.io/instance=es-topology-cluster
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
es-topology-cluster ClusterIP 10.8.11.95 <none> 9200/TCP 4m
es-topology-cluster-master ClusterIP None <none> 9300/TCP 4m
es-topology-cluster-pods ClusterIP None <none> 9200/TCP 4m
Here, we are going to use the es-topology-cluster Service to connect with the database. Now, let’s port-forward the es-topology-cluster Service.
$ kubectl port-forward -n es-demo svc/es-topology-cluster 9200
Forwarding from 127.0.0.1:9200 -> 9200
Forwarding from [::1]:9200 -> 9200
Export the Credentials
Kubernetes Elasticsearch operator will create some Secrets for the database. Let’s list the Secrets for our es-topology-cluster.
$ kubectl get secret -n es-demo -l=app.kubernetes.io/instance=es-topology-cluster
NAME TYPE DATA AGE
es-topology-cluster-apm-system-cred kubernetes.io/basic-auth 2 5m
es-topology-cluster-beats-system-cred kubernetes.io/basic-auth 2 5m
es-topology-cluster-ca-cert kubernetes.io/tls 2 5m
es-topology-cluster-client-cert kubernetes.io/tls 3 5m
es-topology-cluster-config Opaque 1 5m
es-topology-cluster-elastic-cred kubernetes.io/basic-auth 2 5m
es-topology-cluster-http-cert kubernetes.io/tls 3 5m
es-topology-cluster-kibana-system-cred kubernetes.io/basic-auth 2 5m
es-topology-cluster-logstash-system-cred kubernetes.io/basic-auth 2 5m
es-topology-cluster-remote-monitoring-user-cred kubernetes.io/basic-auth 2 5m
es-topology-cluster-transport-cert kubernetes.io/tls 3 5m
Now, we can connect to the database with any of these secret that have the prefix cred. Here, we will use es-topology-cluster-elastic-cred which contains the admin level credentials to connect with the database.
$ kubectl get secret -n es-demo es-topology-cluster-elastic-cred -o jsonpath='{.data.username}' | base64 -d
elastic
$ kubectl get secret -n es-demo es-topology-cluster-elastic-cred -o jsonpath='{.data.password}' | base64 -d
q)UC;l!!euEnk.ZZ
We will now use curl to post some sample data into Elasticsearch. Use the -k flag to disable attempts to verify self-signed certificates for testing purposes.
$ curl -XPOST -k --user 'elastic:q)UC;l!!euEnk.ZZ' "https://localhost:9200/music/_doc?pretty" -H 'Content-Type: application/json' -d'
{
"Artist": "Bon Jovi",
"Song": "Its My Life"
}
'
{
"_index" : "music",
"_id" : "oyO52YYBpvYWEhqZyPph",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1
}
Now, let’s verify that the index has been created successfully.
$ curl -XGET -k --user 'elastic:q)UC;l!!euEnk.ZZ' "https://localhost:9200/_cat/indices?v&s=index&pretty"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open kubedb-system zeIvl0RAQ6uhhcvW5GoWfg 1 1 1 5 834kb 404.9kb
green open music _dAwOpXMSK2eHmc2Un7MNA 1 1 1 0 10.6kb 5.3kb
Also, let’s verify the data in the indexes:
$ curl -XGET -k --user 'elastic:q)UC;l!!euEnk.ZZ' "https://localhost:9200/music/_search?pretty"
{
"took" : 11,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "music",
"_id" : "oyO52YYBpvYWEhqZyPph",
"_score" : 1.0,
"_source" : {
"Artist" : "Bon Jovi",
"Song" : "Its My Life"
}
}
]
}
}
We’ve successfully Deploy Elasticsearch to Kubernetes via Kubernetes Elasticsearch operator KubeDB and insert some sample data into it.
Elasticsearch on Kubernetes: Best Practices
To ensure the stability of your application while using Elasticsearch on Kubernetes using Kubernetes Elasticsearch operator, there are some best practices that you should follow:
Use Dashboard like Kibana: Deploy Kibana alongside Elasticsearch for real-time performance insights and data visualization. Secure Kibana with access controls and encryption, and leverage its dashboards and visualization tools to monitor Elasticsearch cluster health and gain valuable application performance insights.
High Availability: Ensure high availability by utilizing Elasticsearch’s data replication features, distributing data across multiple nodes for redundancy. Kubernetes pod anti-affinity rules help prevent single points of failure, and using a load balancer evenly distributes traffic among nodes.
Backup and Recovery: Prioritize backup and recovery with regular data backups using Elasticsearch snapshots or available backup and recovery solutions. Store backups in separate locations or cloud storage for disaster recovery, and routinely test data restoration processes for reliability.
Monitoring & Security: Implement comprehensive monitoring using tools like Prometheus, Grafana, or Elasticsearch’s monitoring features to track cluster health and performance metrics. Strengthen security with RBAC and robust authentication mechanisms, and implement Kubernetes network policies to Elasticsearch pods.
Conclusion
Elasticsearch is a powerful, open-source search and analytics engine renowned for its ability to handle large and diverse datasets with speed and precision. You have now successfully installed an Elasticsearch database on Kubernetes using the Kubernetes Elasticsearch operator, which is suitable for a variety of use cases. You can visit the official Elasticsearch documentation for further details. Managing database, whether on-premises or in the cloud, requires extensive knowledge and consistent practice. KubeDB provides a full support solution to ensure that your database management fulfills performance and uptime requirements. Whether your database infrastructure is hosted on-site, geographically distributed, or utilizing cloud services or database-as-a-service providers, KubeDB offers assistance in managing the entire process in a production-grade environment.
