Up And Running With Kafka On AWS EKS Using Strimzi

09.10.2019 — kubernetes, aws — 3 min read

Configure the AWS CLI
Create the EKS cluster
Enter Kubernetes
Install and configure Helm
Install the Strimzi Kafka Operator
Deploying the Kafka cluster
Analysis
Test the Kafka cluster with Node.js clients
Clean up!

Let's get right into it, then!

We will be using eksctl, the official CLI for Amazon EKS, to spin up our K8s cluster.

Configure the AWS CLI

Ensure that the AWS CLI is configured. To view your configuration:

1$ aws configure list
2      Name                    Value             Type    Location
3      ----                    -----             ----    --------
4   profile                <not set>             None    None
5access_key     ****************7ONG shared-credentials-file    
6secret_key     ****************lbQg shared-credentials-file    
7    region               ap-south-1      config-file    ~/.aws/config

Note: The aws CLI config and credentials details are usually stored at ~/.aws/config and ~/.aws/credentials respectively.

Create the EKS cluster

1$ eksctl create cluster --name=kafka-eks-cluster --nodes=4 --region=ap-south-1
2
3[ℹ]  using region ap-south-1
4[ℹ]  setting availability zones to [ap-south-1b ap-south-1a ap-south-1c]
5[ℹ]  subnets for ap-south-1b - public:192.168.0.0/19 private:192.168.96.0/19
6[ℹ]  subnets for ap-south-1a - public:192.168.32.0/19 private:192.168.128.0/19
7[ℹ]  subnets for ap-south-1c - public:192.168.64.0/19 private:192.168.160.0/19
8[ℹ]  nodegroup "ng-9f3cbfc7" will use "ami-09c3eb35bb3be46a4" [AmazonLinux2/1.12]
9[ℹ]  creating EKS cluster "kafka-eks-cluster" in "ap-south-1" region
10[ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
11[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-south-1 --name=kafka-eks-cluster'
12[ℹ]  2 sequential tasks: { create cluster control plane "kafka-eks-cluster", create nodegroup "ng-9f3cbfc7" }
13[ℹ]  building cluster stack "eksctl-kafka-eks-cluster-cluster"
14[ℹ]  deploying stack "eksctl-kafka-eks-cluster-cluster"
15[ℹ]  building nodegroup stack "eksctl-kafka-eks-cluster-nodegroup-ng-9f3cbfc7"
16[ℹ]  --nodes-min=4 was set automatically for nodegroup ng-9f3cbfc7
17[ℹ]  --nodes-max=4 was set automatically for nodegroup ng-9f3cbfc7
18[ℹ]  deploying stack "eksctl-kafka-eks-cluster-nodegroup-ng-9f3cbfc7"
19[✔]  all EKS cluster resource for "kafka-eks-cluster" had been created
20[✔]  saved kubeconfig as "/Users/Bensooraj/.kube/config"
21[ℹ]  adding role "arn:aws:iam::account_numer:role/eksctl-kafka-eks-cluster-nodegrou-NodeInstanceRole-IG63RKPE03YQ" to auth ConfigMap
22[ℹ]  nodegroup "ng-9f3cbfc7" has 0 node(s)
23[ℹ]  waiting for at least 4 node(s) to become ready in "ng-9f3cbfc7"
24[ℹ]  nodegroup "ng-9f3cbfc7" has 4 node(s)
25[ℹ]  node "ip-192-168-25-34.ap-south-1.compute.internal" is ready
26[ℹ]  node "ip-192-168-50-249.ap-south-1.compute.internal" is ready
27[ℹ]  node "ip-192-168-62-231.ap-south-1.compute.internal" is ready
28[ℹ]  node "ip-192-168-69-95.ap-south-1.compute.internal" is ready
29[ℹ]  kubectl command should work with "/Users/Bensooraj/.kube/config", try 'kubectl get nodes'
30[✔]  EKS cluster "kafka-eks-cluster" in "ap-south-1" region is ready

A k8s cluster by the name kafka-eks-cluster will be created with 4 nodes (instance type: m5.large) in the Mumbai region (ap-south-1). You can view these in the AWS Console UI as well,

EKS:

CloudFormation UI:

Also, after the cluster is created, the appropriate kubernetes configuration will be added to your kubeconfig file (defaults to ~/.kube/config). The path to the kubeconfig file can be overridden using the --kubeconfig flag.

Enter Kubernetes

Fetching all k8s controllers lists the default kubernetes service. This confirms that kubectl is properly configured to point to the cluster that we just created.

1$ kubectl get all
2NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
3service/kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   19m

Install and configure Helm

Helm is a package manager and application management tool for Kubernetes that packages multiple Kubernetes resources into a single logical deployment unit called Chart.

I use Homebrew, so the installation was pretty straightforward: brew install kubernetes-helm.

Alternatively, to install helm, run the following:

1$ cd ~/eks-kafka-strimzi
2
3$ curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh
4
5$ chmod +x get_helm.sh
6
7$ ./get_helm.sh

Read through their installation guide, if you are looking for more options.

Do not run helm init yet.

Helm relies on a service called tiller that requires special permission on the kubernetes cluster, so we need to build a Service Account (RBAC access) for tiller to use.

The rbac.yaml file would look like the following:

1---
2apiVersion: v1
3kind: ServiceAccount
4metadata:
5  name: tiller
6  namespace: kube-system
7---
8apiVersion: rbac.authorization.k8s.io/v1beta1
9kind: ClusterRoleBinding
10metadata:
11  name: tiller
12roleRef:
13  apiGroup: rbac.authorization.k8s.io
14  kind: ClusterRole
15  name: cluster-admin
16subjects:
17  - kind: ServiceAccount
18    name: tiller
19    namespace: kube-system

Apply this to the kafka-eks-cluster cluster:

1$ kubectl apply -f rbac.yaml
2serviceaccount/tiller created
3clusterrolebinding.rbac.authorization.k8s.io/tiller created
4
5# Verify (listing only the relevant ones)
6$ kubectl get sa,clusterrolebindings --namespace=kube-system
7NAME                        SECRETS   AGE
8.
9serviceaccount/tiller       1         5m22s
10.
11
12NAME                                                                                                AGE
13.
14clusterrolebinding.rbac.authorization.k8s.io/tiller                                                 5m23s
15.

Now, run helm init using the service account we setup. This will install tiller into the cluster which gives it access to manage resources in your cluster.

1$ helm init --service-account=tiller
2
3$HELM_HOME has been configured at /Users/Bensooraj/.helm.
4
5Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
6
7Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
8
9To prevent this, run `helm init` with the --tiller-tls-verify flag.
10
11For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation

Install the Strimzi Kafka Operator

Add the Strimzi repository and install the Strimzi Helm Chart:

1# Add the repo
2$ helm repo add strimzi http://strimzi.io/charts/
3"strimzi" has been added to your repositories
4
5# Search for all Strimzi  charts
6$ helm search strim
7NAME                            CHART VERSION   APP VERSION DESCRIPTION                
8strimzi/strimzi-kafka-operator  0.14.0          0.14.0      Strimzi: Kafka as a Service
9
10# Install the kafka operator
11$ helm install strimzi/strimzi-kafka-operator
12NAME:   bulging-gnat
13LAST DEPLOYED: Wed Oct  2 15:23:45 2019
14NAMESPACE: default
15STATUS: DEPLOYED
16
17RESOURCES:
18==> v1/ClusterRole
19NAME                                 AGE
20strimzi-cluster-operator-global      0s
21strimzi-cluster-operator-namespaced  0s
22strimzi-entity-operator              0s
23strimzi-kafka-broker                 0s
24strimzi-topic-operator               0s
25
26==> v1/ClusterRoleBinding
27NAME                                              AGE
28strimzi-cluster-operator                          0s
29strimzi-cluster-operator-kafka-broker-delegation  0s
30
31==> v1/Deployment
32NAME                      READY  UP-TO-DATE  AVAILABLE  AGE
33strimzi-cluster-operator  0/1    1           0          0s
34
35==> v1/Pod(related)
36NAME                                       READY  STATUS             RESTARTS  AGE
37strimzi-cluster-operator-6667fbc5f8-cqvdv  0/1    ContainerCreating  0         0s
38
39==> v1/RoleBinding
40NAME                                                 AGE
41strimzi-cluster-operator                             0s
42strimzi-cluster-operator-entity-operator-delegation  0s
43strimzi-cluster-operator-topic-operator-delegation   0s
44
45==> v1/ServiceAccount
46NAME                      SECRETS  AGE
47strimzi-cluster-operator  1        0s
48
49==> v1beta1/CustomResourceDefinition
50NAME                                AGE
51kafkabridges.kafka.strimzi.io       0s
52kafkaconnects.kafka.strimzi.io      0s
53kafkaconnects2is.kafka.strimzi.io   0s
54kafkamirrormakers.kafka.strimzi.io  0s
55kafkas.kafka.strimzi.io             1s
56kafkatopics.kafka.strimzi.io        1s
57kafkausers.kafka.strimzi.io         1s
58
59NOTES:
60Thank you for installing strimzi-kafka-operator-0.14.0
61
62To create a Kafka cluster refer to the following documentation.
63
64https://strimzi.io/docs/0.14.0/#kafka-cluster-str

List all the kubernetes objects created again:

1$ kubectl get all
2NAME                                            READY   STATUS    RESTARTS   AGE
3pod/strimzi-cluster-operator-6667fbc5f8-cqvdv   1/1     Running   0          9m25s
4
5NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
6service/kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   90m
7
8NAME                                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
9deployment.apps/strimzi-cluster-operator   1         1         1            1           9m25s
10
11NAME                                                  DESIRED   CURRENT   READY   AGE
12replicaset.apps/strimzi-cluster-operator-6667fbc5f8   1         1         1       9m26s

Deploying the Kafka cluster

We will now create a Kafka cluster with 3 brokers. The YAML file (kafka-cluster.Kafka.yaml) for creating the Kafka cluster would like the following:

1apiVersion: kafka.strimzi.io/v1beta1
2kind: Kafka
3metadata:
4  name: kafka-cluster
5spec:
6  kafka:
7    version: 2.3.0 # Kafka version
8    replicas: 3 # Replicas specifies the number of broker nodes.
9    listeners: # Listeners configure how clients connect to the Kafka cluster
10      plain: {} # 9092
11      tls: {} # 9093
12    config:
13      offsets.topic.replication.factor: 3
14      transaction.state.log.replication.factor: 3
15      transaction.state.log.min.isr: 2
16      log.message.format.version: "2.3"
17      delete.topic.enable: "true"
18    storage:
19      type: persistent-claim
20      size: 10Gi
21      deleteClaim: false
22  zookeeper:
23    replicas: 3
24    storage:
25      type: persistent-claim # Persistent storage backed by AWS EBS
26      size: 10Gi
27      deleteClaim: false
28  entityOperator:
29    topicOperator: {} # Operator for topic administration
30    userOperator: {}

Apply the above YAML file:

1$ kubectl apply -f kafka-cluster.Kafka.yaml

Analysis

This is where things get interesting. We will now analyse some of the k8s resources which the strimzi kafka operator has created for us under the hood.

1$ kubectl get statefulsets.apps,pod,deployments,svc
2NAME                                       DESIRED   CURRENT   AGE
3statefulset.apps/kafka-cluster-kafka       3         3         78m
4statefulset.apps/kafka-cluster-zookeeper   3         3         79m
5
6NAME                                                 READY   STATUS    RESTARTS   AGE
7pod/kafka-cluster-entity-operator-54cb77fd9d-9zbcx   3/3     Running   0          77m
8pod/kafka-cluster-kafka-0                            2/2     Running   0          78m
9pod/kafka-cluster-kafka-1                            2/2     Running   0          78m
10pod/kafka-cluster-kafka-2                            2/2     Running   0          78m
11pod/kafka-cluster-zookeeper-0                        2/2     Running   0          79m
12pod/kafka-cluster-zookeeper-1                        2/2     Running   0          79m
13pod/kafka-cluster-zookeeper-2                        2/2     Running   0          79m
14pod/strimzi-cluster-operator-6667fbc5f8-cqvdv        1/1     Running   0          172m
15
16NAME                                                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
17deployment.extensions/kafka-cluster-entity-operator   1         1         1            1           77m
18deployment.extensions/strimzi-cluster-operator        1         1         1            1           172m
19
20NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
21service/kafka-cluster-kafka-bootstrap    ClusterIP   10.100.177.177   <none>        9091/TCP,9092/TCP,9093/TCP   78m
22service/kafka-cluster-kafka-brokers      ClusterIP   None             <none>        9091/TCP,9092/TCP,9093/TCP   78m
23service/kafka-cluster-zookeeper-client   ClusterIP   10.100.199.128   <none>        2181/TCP                     79m
24service/kafka-cluster-zookeeper-nodes    ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP   79m
25service/kubernetes                       ClusterIP   10.100.0.1       <none>        443/TCP                      4h13m

Points to note:

The StatefulSet kafka-cluster-zookeeper has created 3 pods - kafka-cluster-zookeeper-0, kafka-cluster-zookeeper-1 and kafka-cluster-zookeeper-2. The headless service kafka-cluster-zookeeper-nodes facilitates network identity of these 3 pods (the 3 Zookeeper nodes).
The StatefulSet kafka-cluster-kafka has created 3 pods - kafka-cluster-kafka-0, kafka-cluster-kafka-1 and kafka-cluster-kafka-2. The headless service kafka-cluster-kafka-brokers facilitates network identity of these 3 pods (the 3 Kafka brokers).

Persistent volumes are dynamically provisioned:

1$ kubectl get pv,pvc
2NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                    STORAGECLASS   REASON   AGE
3persistentvolume/pvc-7ff2909f-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            Delete           Bound    default/data-kafka-cluster-zookeeper-1   gp2                     11h
4persistentvolume/pvc-7ff290c4-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            Delete           Bound    default/data-kafka-cluster-zookeeper-2   gp2                     11h
5persistentvolume/pvc-7ffd1d22-e507-11e9-a775-029ce0835b96   10Gi       RWO            Delete           Bound    default/data-kafka-cluster-zookeeper-0   gp2                     11h
6persistentvolume/pvc-a5997b77-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            Delete           Bound    default/data-kafka-cluster-kafka-0       gp2                     11h
7persistentvolume/pvc-a599e52b-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            Delete           Bound    default/data-kafka-cluster-kafka-1       gp2                     11h
8persistentvolume/pvc-a59c6cd2-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            Delete           Bound    default/data-kafka-cluster-kafka-2       gp2                     11h
9
10NAME                                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
11persistentvolumeclaim/data-kafka-cluster-kafka-0       Bound    pvc-a5997b77-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            gp2            11h
12persistentvolumeclaim/data-kafka-cluster-kafka-1       Bound    pvc-a599e52b-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            gp2            11h
13persistentvolumeclaim/data-kafka-cluster-kafka-2       Bound    pvc-a59c6cd2-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            gp2            11h
14persistentvolumeclaim/data-kafka-cluster-zookeeper-0   Bound    pvc-7ffd1d22-e507-11e9-a775-029ce0835b96   10Gi       RWO            gp2            11h
15persistentvolumeclaim/data-kafka-cluster-zookeeper-1   Bound    pvc-7ff2909f-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            gp2            11h
16persistentvolumeclaim/data-kafka-cluster-zookeeper-2   Bound    pvc-7ff290c4-e507-11e9-91df-0a1e73fdd786   10Gi       RWO            gp2            11h

You can view the provisioned AWS EBS volumes in the UI as well:

Create topics

Before we get started with clients we need to create a topic (with 3 partitions and a replication factor of 3), over which our producer and the consumer and produce messages and consume messages on respectively.

1apiVersion: kafka.strimzi.io/v1beta1
2kind: KafkaTopic
3metadata:
4  name: test-topic
5  labels:
6    strimzi.io/cluster: kafka-cluster
7spec:
8  partitions: 3
9  replicas: 3

Apply the YAML to the k8s cluster:

1$ kubectl apply -f create-topics.yaml
2kafkatopic.kafka.strimzi.io/test-topic created

Test the Kafka cluster with Node.js clients

The multi-broker Kafka cluster that we deployed is backed by statefulsets and their corresponding headless services.

Since each Pod (Kafka broker) now has a network identity, clients can connect to the Kafka brokers via a combination of the pod name and service name: $(podname).$(governing service domain). In our case, these would be the following URLs:

kafka-cluster-kafka-0.kafka-cluster-kafka-brokers
kafka-cluster-kafka-1.kafka-cluster-kafka-brokers
kafka-cluster-kafka-2.kafka-cluster-kafka-brokers

Note:

If the Kafka cluster is deployed in a different namespace, you will have to expand it a little further: $(podname).$(service name).$(namespace).svc.cluster.local.
Alternatively, the clients can connect to the Kafka cluster using the Service kafka-cluster-kafka-bootstrap:9092 as well. It distributes the connection over the three broker specific endpoints I have listed above. As I no longer keep track of the individual broker endpoints, this method plays out well when I have to scale up or down the number of brokers in the Kafka cluster.

First, clone this repo: {% github bensooraj/strimzi-kafka-aws-eks no-readme %}

1# Create the configmap, which contains details such as the broker DNS names, topic name and consumer group ID
2$ kubectl apply -f test/k8s/config.yaml
3configmap/kafka-client-config created
4
5# Create the producer deployment
6$ kubectl apply -f test/k8s/producer.Deployment.yaml
7deployment.apps/node-test-producer created
8
9# Expose the producer deployment via a service of type LoadBalancer (backed by the AWS Elastic Load Balancer). This just makes it easy for me to curl from postman
10$ kubectl apply -f test/k8s/producer.Service.yaml
11service/node-test-producer created
12
13# Finally, create the consumer deployment
14$ kubectl apply -f test/k8s/consumer.Deployment.yaml
15deployment.apps/node-test-consumer created

If you list the producer service that we created, you would notice a URL under EXTERNAL-IP:

1$ kubectl get svc
2NAME                             TYPE           CLUSTER-IP       EXTERNAL-IP                                                                PORT(S)                      AGE
3.
4.
5node-test-producer               LoadBalancer   10.100.145.203   ac5f3d0d1e55a11e9a775029ce0835b9-2040242746.ap-south-1.elb.amazonaws.com   80:31231/TCP                 55m

The URL ac5f3d0d1e55a11e9a775029ce0835b9-2040242746.ap-south-1.elb.amazonaws.com is an AWS ELB backed public endpoint which we will be querying for producing messages to the Kafka cluster.

Also, you can see that there is 1 producer and 3 consumers (one for each partition of the topic test-topic):

1$ kubectl get pod
2NAME                                             READY   STATUS    RESTARTS   AGE
3node-test-consumer-96b44cbcb-gs2km               1/1     Running   0          125m
4node-test-consumer-96b44cbcb-ptvjd               1/1     Running   0          125m
5node-test-consumer-96b44cbcb-xk75j               1/1     Running   0          125m
6node-test-producer-846d9c5986-vcsf2              1/1     Running   0          125m

The producer app basically exposes 3 URLs:

/kafka-test/green/:message
/kafka-test/blue/:message
/kafka-test/cyan/:message

Where :message can be any valid string. Each of these URLs produce a message along with the colour information to the topic test-topic.

The consumer group (the 3 consumer pods that we spin-up) listening for any incoming messages from the topic test-topic, then receives these messages and prints them on to the console according to the colour instruction.

I curl each URL 3 times. From the following GIF you can see how message consumption is distributed across the 3 consumers in a round-robin manner:

Clean Up!

1# Delete the test producer and consumer apps:
2$ kubectl delete -f test/k8s/
3configmap "kafka-client-config" deleted
4deployment.apps "node-test-consumer" deleted
5deployment.apps "node-test-producer" deleted
6service "node-test-producer" deleted
7
8# Delete the Kafka cluster
9$ kubectl delete kafka kafka-cluster
10kafka.kafka.strimzi.io "kafka-cluster" deleted
11
12# Delete the Strimzi cluster operator
13$ kubectl delete deployments. strimzi-cluster-operator
14deployment.extensions "strimzi-cluster-operator" deleted
15
16# Manually delete the persistent volumes
17# Kafka
18$ kubectl delete pvc data-kafka-cluster-kafka-0
19$ kubectl delete pvc data-kafka-cluster-kafka-1
20$ kubectl delete pvc data-kafka-cluster-kafka-2
21# Zookeeper
22$ kubectl delete pvc data-kafka-cluster-zookeeper-0
23$ kubectl delete pvc data-kafka-cluster-zookeeper-1
24$ kubectl delete pvc data-kafka-cluster-zookeeper-2

Finally, delete the EKS cluster:

1$ eksctl delete cluster kafka-eks-cluster
2[ℹ]  using region ap-south-1
3[ℹ]  deleting EKS cluster "kafka-eks-cluster"
4[✔]  kubeconfig has been updated
5[ℹ]  2 sequential tasks: { delete nodegroup "ng-9f3cbfc7", delete cluster control plane "kafka-eks-cluster" [async] }
6[ℹ]  will delete stack "eksctl-kafka-eks-cluster-nodegroup-ng-9f3cbfc7"
7[ℹ]  waiting for stack "eksctl-kafka-eks-cluster-nodegroup-ng-9f3cbfc7" to get deleted
8[ℹ]  will delete stack "eksctl-kafka-eks-cluster-cluster"
9[✔]  all cluster resources were deleted

Hope this helped!

Note: This is not a tutorial per se, instead, this is me recording my observations as I setup a Kafka cluster for the first time on a Kubernetes platform using Strimzi.