New to KubeDB? Please start here.

Run Kafka with TLS/SSL (Transport Encryption)

KubeDB supports providing TLS/SSL encryption for Kafka. This tutorial will show you how to use KubeDB to run a Kafka cluster with TLS/SSL encryption.

Before You Begin

At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using kind.
Install cert-manger v1.0.0 or later to your cluster to manage your SSL/TLS certificates.
Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps here.
To keep things isolated, this tutorial uses a separate namespace called demo throughout this tutorial.
```
$ kubectl create ns demo
namespace/demo created
```

Note: YAML files used in this tutorial are stored in docs/examples/kafka folder in GitHub repository kubedb/docs.

Overview

KubeDB uses following crd fields to enable SSL/TLS encryption in Kafka.

spec:
- enableSSL
- tls:
  - issuerRef
  - certificate

Read about the fields in details in kafka concept,

tls is applicable for all types of Kafka (i.e., combined and topology).

Users must specify the tls.issuerRef field. KubeDB uses the issuer or clusterIssuer referenced in the tls.issuerRef field, and the certificate specs provided in tls.certificate to generate certificate secrets. These certificate secrets are then used to generate required certificates including ca.crt, tls.crt, tls.key, keystore.jks and truststore.jks.

Create Issuer/ ClusterIssuer

We are going to create an example Issuer that will be used throughout the duration of this tutorial to enable SSL/TLS in Kafka. Alternatively, you can follow this cert-manager tutorial to create your own Issuer.

Start off by generating you ca certificates using openssl.

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=kafka/O=kubedb"

Now create a ca-secret using the certificate files you have just generated.

kubectl create secret tls kafka-ca \
     --cert=ca.crt \
     --key=ca.key \
     --namespace=demo

Now, create an Issuer using the ca-secret you have just created. The YAML file looks like this:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: kafka-ca-issuer
  namespace: demo
spec:
  ca:
    secretName: kafka-ca

Apply the YAML file:

$ kubectl create -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/examples/kafka/tls/kf-issuer.yaml
issuer.cert-manager.io/kafka-ca-issuer created

TLS/SSL encryption in Kafka Combined Cluster

apiVersion: kubedb.com/v1
kind: Kafka
metadata:
  name: kafka-dev-tls
  namespace: demo
spec:
  version: 3.6.1
  enableSSL: true
  tls:
    issuerRef:
      apiGroup: "cert-manager.io"
      kind: Issuer
      name: kafka-ca-issuer
  replicas: 3
  storage:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
    storageClassName: standard
  storageType: Durable
  deletionPolicy: WipeOut

Deploy Kafka Combined Cluster

$ kubectl create -f https://github.com/kubedb/docs/raw/v2024.11.18/docs/examples/kafka/tls/kafka-dev-tls.yaml
kafka.kubedb.com/kafka-dev-tls created

Now, wait until kafka-dev-tls created has status Ready. i.e,

$ watch kubectl get mg -n demo

Every 2.0s: kubectl get kafka -n demo                                                                                                                          aadee: Fri Sep  6 12:34:51 2024
NAME            TYPE            VERSION   STATUS         AGE
kafka-dev-tls   kubedb.com/v1   3.6.1     Provisioning   0s
kafka-dev-tls   kubedb.com/v1   3.6.1     Provisioning   12s
.
.
kafka-dev-tls   kubedb.com/v1   3.6.1     Ready          77s

Verify TLS/SSL in Kafka Combined Cluster

$ kubectl describe secret -n demo kafka-dev-tls-client-cert

Name:         kafka-dev-tls-client-cert
Namespace:    demo
Labels:       app.kubernetes.io/component=database
              app.kubernetes.io/instance=kafka-dev-tls
              app.kubernetes.io/managed-by=kubedb.com
              app.kubernetes.io/name=kafkas.kubedb.com
              controller.cert-manager.io/fao=true
Annotations:  cert-manager.io/alt-names:
                *.kafka-dev-tls-pods.demo.svc.cluster.local,kafka-dev-tls-pods,kafka-dev-tls-pods.demo.svc,kafka-dev-tls-pods.demo.svc.cluster.local,local...
              cert-manager.io/certificate-name: kafka-dev-tls-client-cert
              cert-manager.io/common-name: kafka-dev-tls-pods.demo.svc
              cert-manager.io/ip-sans: 127.0.0.1
              cert-manager.io/issuer-group: cert-manager.io
              cert-manager.io/issuer-kind: Issuer
              cert-manager.io/issuer-name: kafka-ca-issuer
              cert-manager.io/uri-sans: 

Type:  kubernetes.io/tls

Data
====
truststore.jks:  891 bytes
ca.crt:          1184 bytes
keystore.jks:    3245 bytes
tls.crt:         1452 bytes
tls.key:         1704 bytes

Now, Let’s exec into a kafka broker pod and verify the configuration that the TLS is enabled.

$ kubectl exec -it -n demo kafka-dev-tls-0 -- kafka-configs.sh --bootstrap-server localhost:9092 --command-config /opt/kafka/config/clientauth.properties --describe --entity-type brokers --all | grep 'ssl.keystore'
  ssl.keystore.certificate.chain=null sensitive=true synonyms={}
  ssl.keystore.key=null sensitive=true synonyms={}
  ssl.keystore.location=/var/private/ssl/server.keystore.jks sensitive=false synonyms={STATIC_BROKER_CONFIG:ssl.keystore.location=/var/private/ssl/server.keystore.jks}
  ssl.keystore.password=null sensitive=true synonyms={STATIC_BROKER_CONFIG:ssl.keystore.password=null}
  ssl.keystore.type=JKS sensitive=false synonyms={DEFAULT_CONFIG:ssl.keystore.type=JKS}
  zookeeper.ssl.keystore.location=null sensitive=false synonyms={}
  zookeeper.ssl.keystore.password=null sensitive=true synonyms={}
  zookeeper.ssl.keystore.type=null sensitive=false synonyms={}
  ssl.keystore.certificate.chain=null sensitive=true synonyms={}
  ssl.keystore.key=null sensitive=true synonyms={}
  ssl.keystore.location=/var/private/ssl/server.keystore.jks sensitive=false synonyms={STATIC_BROKER_CONFIG:ssl.keystore.location=/var/private/ssl/server.keystore.jks}
  ssl.keystore.password=null sensitive=true synonyms={STATIC_BROKER_CONFIG:ssl.keystore.password=null}
  ssl.keystore.type=JKS sensitive=false synonyms={DEFAULT_CONFIG:ssl.keystore.type=JKS}
  zookeeper.ssl.keystore.location=null sensitive=false synonyms={}
  zookeeper.ssl.keystore.password=null sensitive=true synonyms={}
  zookeeper.ssl.keystore.type=null sensitive=false synonyms={}
  ssl.keystore.certificate.chain=null sensitive=true synonyms={}
  ssl.keystore.key=null sensitive=true synonyms={}
  ssl.keystore.location=/var/private/ssl/server.keystore.jks sensitive=false synonyms={STATIC_BROKER_CONFIG:ssl.keystore.location=/var/private/ssl/server.keystore.jks}
  ssl.keystore.password=null sensitive=true synonyms={STATIC_BROKER_CONFIG:ssl.keystore.password=null}
  ssl.keystore.type=JKS sensitive=false synonyms={DEFAULT_CONFIG:ssl.keystore.type=JKS}
  zookeeper.ssl.keystore.location=null sensitive=false synonyms={}
  zookeeper.ssl.keystore.password=null sensitive=true synonyms={}
  zookeeper.ssl.keystore.type=null sensitive=false synonyms={}

We can see from the above output that, keystore location is /var/private/ssl/server.keystore.jks which means that TLS is enabled.

You will find a file named clientauth.properties in the config directory. This file is generated by the operator which contains necessary authentication/authorization/certificate configurations that are required during connect to the Kafka cluster.

root@kafka-dev-tls-0:~# cat config/clientauth.properties
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="*************";
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
ssl.truststore.location=/var/private/ssl/server.truststore.jks
ssl.truststore.password=***********

Now, let’s exec into the kafka pod and connect using this configuration to verify the TLS is enabled.

$ kubectl exec -it -n demo kafka-dev-tls-0 -- bash
kafka@kafka-dev-tls-0:~$ kafka-metadata-quorum.sh --command-config config/clientauth.properties --bootstrap-server localhost:9092 describe --status
ClusterId:              11ef-921c-f2a07f85765w
LeaderId:               1
LeaderEpoch:            17
HighWatermark:          1292
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   16
CurrentVoters:          [0,1,2]
CurrentObservers:       []

From the above output, we can see that we are able to connect to the Kafka cluster using the TLS configuration.

Cleaning up

To cleanup the Kubernetes resources created by this tutorial, run:

kubectl delete kafka -n demo kafka-dev-tls
kubectl delete issuer -n demo kafka-ca-issuer
kubectl delete ns demo

Next Steps

Detail concepts of Kafka object.
Monitor your Kafka cluster with KubeDB using out-of-the-box Prometheus operator.
Monitor your Kafka cluster with KubeDB using out-of-the-box builtin-Prometheus.
Use kubedb cli to manage databases like kubectl for Kubernetes.
Detail concepts of Kafka object.
Want to hack on KubeDB? Check our contribution guidelines.

KubeDB

Stash

KubeVault

Voyager

ConfigSyncer

Guard

RESOURCES

Druid

Elasticsearch

FerretDB

Kafka

MariaDB

Memcached

Microsoft SQL Server

MongoDB

MySQL

OpenSearch

PerconaXtraDB

PgBouncer

Pgpool

PostgreSQL

ProxySQL

RabbitMQ

Redis

SingleStore

Solr

ZooKeeper