You are looking at the documentation of a prior release. To read the documentation of the latest release, please
visit here.
New to KubeDB? Please start here.
Run Kafka with TLS/SSL (Transport Encryption)
KubeDB supports providing TLS/SSL encryption for Kafka. This tutorial will show you how to use KubeDB to run a Kafka cluster with TLS/SSL encryption.
Before You Begin
At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using kind.
Install
cert-manger
v1.0.0 or later to your cluster to manage your SSL/TLS certificates.Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps here.
To keep things isolated, this tutorial uses a separate namespace called
demo
throughout this tutorial.$ kubectl create ns demo namespace/demo created
Note: YAML files used in this tutorial are stored in docs/examples/kafka folder in GitHub repository kubedb/docs.
Overview
KubeDB uses following crd fields to enable SSL/TLS encryption in Kafka.
spec:
enableSSL
tls:
issuerRef
certificate
Read about the fields in details in kafka concept,
tls
is applicable for all types of Kafka (i.e., combined
and topology
).
Users must specify the tls.issuerRef
field. KubeDB uses the issuer
or clusterIssuer
referenced in the tls.issuerRef
field, and the certificate specs provided in tls.certificate
to generate certificate secrets. These certificate secrets are then used to generate required certificates including ca.crt
, tls.crt
, tls.key
, keystore.jks
and truststore.jks
.
Create Issuer/ ClusterIssuer
We are going to create an example Issuer
that will be used throughout the duration of this tutorial to enable SSL/TLS in Kafka. Alternatively, you can follow this cert-manager tutorial to create your own Issuer
.
- Start off by generating you ca certificates using openssl.
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=kafka/O=kubedb"
- Now create a ca-secret using the certificate files you have just generated.
kubectl create secret tls kafka-ca \
--cert=ca.crt \
--key=ca.key \
--namespace=demo
Now, create an Issuer
using the ca-secret
you have just created. The YAML
file looks like this:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: kafka-ca-issuer
namespace: demo
spec:
ca:
secretName: kafka-ca
Apply the YAML
file:
$ kubectl create -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/examples/kafka/tls/kf-issuer.yaml
issuer.cert-manager.io/kafka-ca-issuer created
TLS/SSL encryption in Kafka Combined Cluster
apiVersion: kubedb.com/v1
kind: Kafka
metadata:
name: kafka-dev-tls
namespace: demo
spec:
version: 3.6.1
enableSSL: true
tls:
issuerRef:
apiGroup: "cert-manager.io"
kind: Issuer
name: kafka-ca-issuer
replicas: 3
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: standard
storageType: Durable
deletionPolicy: WipeOut
Deploy Kafka Combined Cluster
$ kubectl create -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/examples/kafka/tls/kafka-dev-tls.yaml
kafka.kubedb.com/kafka-dev-tls created
Now, wait until kafka-dev-tls created
has status Ready
. i.e,
$ watch kubectl get mg -n demo
Every 2.0s: kubectl get kafka -n demo aadee: Fri Sep 6 12:34:51 2024
NAME TYPE VERSION STATUS AGE
kafka-dev-tls kubedb.com/v1 3.6.1 Provisioning 0s
kafka-dev-tls kubedb.com/v1 3.6.1 Provisioning 12s
.
.
kafka-dev-tls kubedb.com/v1 3.6.1 Ready 77s
Verify TLS/SSL in Kafka Combined Cluster
$ kubectl describe secret -n demo kafka-dev-tls-client-cert
Name: kafka-dev-tls-client-cert
Namespace: demo
Labels: app.kubernetes.io/component=database
app.kubernetes.io/instance=kafka-dev-tls
app.kubernetes.io/managed-by=kubedb.com
app.kubernetes.io/name=kafkas.kubedb.com
controller.cert-manager.io/fao=true
Annotations: cert-manager.io/alt-names:
*.kafka-dev-tls-pods.demo.svc.cluster.local,kafka-dev-tls-pods,kafka-dev-tls-pods.demo.svc,kafka-dev-tls-pods.demo.svc.cluster.local,local...
cert-manager.io/certificate-name: kafka-dev-tls-client-cert
cert-manager.io/common-name: kafka-dev-tls-pods.demo.svc
cert-manager.io/ip-sans: 127.0.0.1
cert-manager.io/issuer-group: cert-manager.io
cert-manager.io/issuer-kind: Issuer
cert-manager.io/issuer-name: kafka-ca-issuer
cert-manager.io/uri-sans:
Type: kubernetes.io/tls
Data
====
truststore.jks: 891 bytes
ca.crt: 1184 bytes
keystore.jks: 3245 bytes
tls.crt: 1452 bytes
tls.key: 1704 bytes
Now, Let’s exec into a kafka broker pod and verify the configuration that the TLS is enabled.
$ kubectl exec -it -n demo kafka-dev-tls-0 -- kafka-configs.sh --bootstrap-server localhost:9092 --command-config /opt/kafka/config/clientauth.properties --describe --entity-type brokers --all | grep 'ssl.keystore'
ssl.keystore.certificate.chain=null sensitive=true synonyms={}
ssl.keystore.key=null sensitive=true synonyms={}
ssl.keystore.location=/var/private/ssl/server.keystore.jks sensitive=false synonyms={STATIC_BROKER_CONFIG:ssl.keystore.location=/var/private/ssl/server.keystore.jks}
ssl.keystore.password=null sensitive=true synonyms={STATIC_BROKER_CONFIG:ssl.keystore.password=null}
ssl.keystore.type=JKS sensitive=false synonyms={DEFAULT_CONFIG:ssl.keystore.type=JKS}
zookeeper.ssl.keystore.location=null sensitive=false synonyms={}
zookeeper.ssl.keystore.password=null sensitive=true synonyms={}
zookeeper.ssl.keystore.type=null sensitive=false synonyms={}
ssl.keystore.certificate.chain=null sensitive=true synonyms={}
ssl.keystore.key=null sensitive=true synonyms={}
ssl.keystore.location=/var/private/ssl/server.keystore.jks sensitive=false synonyms={STATIC_BROKER_CONFIG:ssl.keystore.location=/var/private/ssl/server.keystore.jks}
ssl.keystore.password=null sensitive=true synonyms={STATIC_BROKER_CONFIG:ssl.keystore.password=null}
ssl.keystore.type=JKS sensitive=false synonyms={DEFAULT_CONFIG:ssl.keystore.type=JKS}
zookeeper.ssl.keystore.location=null sensitive=false synonyms={}
zookeeper.ssl.keystore.password=null sensitive=true synonyms={}
zookeeper.ssl.keystore.type=null sensitive=false synonyms={}
ssl.keystore.certificate.chain=null sensitive=true synonyms={}
ssl.keystore.key=null sensitive=true synonyms={}
ssl.keystore.location=/var/private/ssl/server.keystore.jks sensitive=false synonyms={STATIC_BROKER_CONFIG:ssl.keystore.location=/var/private/ssl/server.keystore.jks}
ssl.keystore.password=null sensitive=true synonyms={STATIC_BROKER_CONFIG:ssl.keystore.password=null}
ssl.keystore.type=JKS sensitive=false synonyms={DEFAULT_CONFIG:ssl.keystore.type=JKS}
zookeeper.ssl.keystore.location=null sensitive=false synonyms={}
zookeeper.ssl.keystore.password=null sensitive=true synonyms={}
zookeeper.ssl.keystore.type=null sensitive=false synonyms={}
We can see from the above output that, keystore location is /var/private/ssl/server.keystore.jks
which means that TLS is enabled.
You will find a file named clientauth.properties
in the config directory. This file is generated by the operator which contains necessary authentication/authorization/certificate configurations that are required during connect to the Kafka cluster.
root@kafka-dev-tls-0:~# cat config/clientauth.properties
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="*************";
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
ssl.truststore.location=/var/private/ssl/server.truststore.jks
ssl.truststore.password=***********
Now, let’s exec into the kafka pod and connect using this configuration to verify the TLS is enabled.
$ kubectl exec -it -n demo kafka-dev-tls-0 -- bash
kafka@kafka-dev-tls-0:~$ kafka-metadata-quorum.sh --command-config config/clientauth.properties --bootstrap-server localhost:9092 describe --status
ClusterId: 11ef-921c-f2a07f85765w
LeaderId: 1
LeaderEpoch: 17
HighWatermark: 1292
MaxFollowerLag: 0
MaxFollowerLagTimeMs: 16
CurrentVoters: [0,1,2]
CurrentObservers: []
From the above output, we can see that we are able to connect to the Kafka cluster using the TLS configuration.
Cleaning up
To cleanup the Kubernetes resources created by this tutorial, run:
kubectl delete kafka -n demo kafka-dev-tls
kubectl delete issuer -n demo kafka-ca-issuer
kubectl delete ns demo
Next Steps
- Detail concepts of Kafka object.
- Monitor your Kafka cluster with KubeDB using out-of-the-box Prometheus operator.
- Monitor your Kafka cluster with KubeDB using out-of-the-box builtin-Prometheus.
- Use kubedb cli to manage databases like kubectl for Kubernetes.
- Detail concepts of Kafka object.
- Want to hack on KubeDB? Check our contribution guidelines.