— kubernetes, aws — 3 min read
Let's get right into it, then!
We will be using eksctl
, the official CLI for Amazon EKS, to spin up our K8s cluster.
Ensure that the AWS CLI is configured. To view your configuration:
1$ aws configure list2 Name Value Type Location3 ---- ----- ---- --------4 profile <not set> None None5access_key ****************7ONG shared-credentials-file 6secret_key ****************lbQg shared-credentials-file 7 region ap-south-1 config-file ~/.aws/config
Note: The aws CLI config and credentials details are usually stored at ~/.aws/config
and ~/.aws/credentials
respectively.
1$ eksctl create cluster --name=kafka-eks-cluster --nodes=4 --region=ap-south-12
3[ℹ] using region ap-south-14[ℹ] setting availability zones to [ap-south-1b ap-south-1a ap-south-1c]5[ℹ] subnets for ap-south-1b - public:192.168.0.0/19 private:192.168.96.0/196[ℹ] subnets for ap-south-1a - public:192.168.32.0/19 private:192.168.128.0/197[ℹ] subnets for ap-south-1c - public:192.168.64.0/19 private:192.168.160.0/198[ℹ] nodegroup "ng-9f3cbfc7" will use "ami-09c3eb35bb3be46a4" [AmazonLinux2/1.12]9[ℹ] creating EKS cluster "kafka-eks-cluster" in "ap-south-1" region10[ℹ] will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup11[ℹ] if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-south-1 --name=kafka-eks-cluster'12[ℹ] 2 sequential tasks: { create cluster control plane "kafka-eks-cluster", create nodegroup "ng-9f3cbfc7" }13[ℹ] building cluster stack "eksctl-kafka-eks-cluster-cluster"14[ℹ] deploying stack "eksctl-kafka-eks-cluster-cluster"15[ℹ] building nodegroup stack "eksctl-kafka-eks-cluster-nodegroup-ng-9f3cbfc7"16[ℹ] --nodes-min=4 was set automatically for nodegroup ng-9f3cbfc717[ℹ] --nodes-max=4 was set automatically for nodegroup ng-9f3cbfc718[ℹ] deploying stack "eksctl-kafka-eks-cluster-nodegroup-ng-9f3cbfc7"19[✔] all EKS cluster resource for "kafka-eks-cluster" had been created20[✔] saved kubeconfig as "/Users/Bensooraj/.kube/config"21[ℹ] adding role "arn:aws:iam::account_numer:role/eksctl-kafka-eks-cluster-nodegrou-NodeInstanceRole-IG63RKPE03YQ" to auth ConfigMap22[ℹ] nodegroup "ng-9f3cbfc7" has 0 node(s)23[ℹ] waiting for at least 4 node(s) to become ready in "ng-9f3cbfc7"24[ℹ] nodegroup "ng-9f3cbfc7" has 4 node(s)25[ℹ] node "ip-192-168-25-34.ap-south-1.compute.internal" is ready26[ℹ] node "ip-192-168-50-249.ap-south-1.compute.internal" is ready27[ℹ] node "ip-192-168-62-231.ap-south-1.compute.internal" is ready28[ℹ] node "ip-192-168-69-95.ap-south-1.compute.internal" is ready29[ℹ] kubectl command should work with "/Users/Bensooraj/.kube/config", try 'kubectl get nodes'30[✔] EKS cluster "kafka-eks-cluster" in "ap-south-1" region is ready
A k8s cluster by the name kafka-eks-cluster will be created with 4 nodes (instance type: m5.large) in the Mumbai region (ap-south-1). You can view these in the AWS Console UI as well,
EKS:
CloudFormation UI:
Also, after the cluster is created, the appropriate kubernetes configuration will be added to your kubeconfig file (defaults to ~/.kube/config
). The path to the kubeconfig file can be overridden using the --kubeconfig
flag.
Fetching all k8s controllers lists the default kubernetes
service. This confirms that kubectl
is properly configured to point to the cluster that we just created.
1$ kubectl get all2NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE3service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 19m
Helm is a package manager and application management tool for Kubernetes that packages multiple Kubernetes resources into a single logical deployment unit called Chart.
I use Homebrew, so the installation was pretty straightforward: brew install kubernetes-helm
.
Alternatively, to install helm
, run the following:
1$ cd ~/eks-kafka-strimzi2
3$ curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh4
5$ chmod +x get_helm.sh6
7$ ./get_helm.sh
Read through their installation guide, if you are looking for more options.
Do not run helm init
yet.
Helm
relies on a service called tiller
that requires special permission on the kubernetes cluster, so we need to build a Service Account
(RBAC access) for tiller
to use.
The rbac.yaml
file would look like the following:
1---2apiVersion: v13kind: ServiceAccount4metadata:5 name: tiller6 namespace: kube-system7---8apiVersion: rbac.authorization.k8s.io/v1beta19kind: ClusterRoleBinding10metadata:11 name: tiller12roleRef:13 apiGroup: rbac.authorization.k8s.io14 kind: ClusterRole15 name: cluster-admin16subjects:17 - kind: ServiceAccount18 name: tiller19 namespace: kube-system
Apply this to the kafka-eks-cluster
cluster:
1$ kubectl apply -f rbac.yaml2serviceaccount/tiller created3clusterrolebinding.rbac.authorization.k8s.io/tiller created4
5# Verify (listing only the relevant ones)6$ kubectl get sa,clusterrolebindings --namespace=kube-system7NAME SECRETS AGE8.9serviceaccount/tiller 1 5m22s10.11
12NAME AGE13.14clusterrolebinding.rbac.authorization.k8s.io/tiller 5m23s15.
Now, run helm init
using the service account we setup. This will install tiller into the cluster which gives it access to manage resources in your cluster.
1$ helm init --service-account=tiller2
3$HELM_HOME has been configured at /Users/Bensooraj/.helm.4
5Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.6
7Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.8
9To prevent this, run `helm init` with the --tiller-tls-verify flag.10
11For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Add the Strimzi repository and install the Strimzi Helm Chart:
1# Add the repo2$ helm repo add strimzi http://strimzi.io/charts/3"strimzi" has been added to your repositories4
5# Search for all Strimzi charts6$ helm search strim7NAME CHART VERSION APP VERSION DESCRIPTION 8strimzi/strimzi-kafka-operator 0.14.0 0.14.0 Strimzi: Kafka as a Service9
10# Install the kafka operator11$ helm install strimzi/strimzi-kafka-operator12NAME: bulging-gnat13LAST DEPLOYED: Wed Oct 2 15:23:45 201914NAMESPACE: default15STATUS: DEPLOYED16
17RESOURCES:18==> v1/ClusterRole19NAME AGE20strimzi-cluster-operator-global 0s21strimzi-cluster-operator-namespaced 0s22strimzi-entity-operator 0s23strimzi-kafka-broker 0s24strimzi-topic-operator 0s25
26==> v1/ClusterRoleBinding27NAME AGE28strimzi-cluster-operator 0s29strimzi-cluster-operator-kafka-broker-delegation 0s30
31==> v1/Deployment32NAME READY UP-TO-DATE AVAILABLE AGE33strimzi-cluster-operator 0/1 1 0 0s34
35==> v1/Pod(related)36NAME READY STATUS RESTARTS AGE37strimzi-cluster-operator-6667fbc5f8-cqvdv 0/1 ContainerCreating 0 0s38
39==> v1/RoleBinding40NAME AGE41strimzi-cluster-operator 0s42strimzi-cluster-operator-entity-operator-delegation 0s43strimzi-cluster-operator-topic-operator-delegation 0s44
45==> v1/ServiceAccount46NAME SECRETS AGE47strimzi-cluster-operator 1 0s48
49==> v1beta1/CustomResourceDefinition50NAME AGE51kafkabridges.kafka.strimzi.io 0s52kafkaconnects.kafka.strimzi.io 0s53kafkaconnects2is.kafka.strimzi.io 0s54kafkamirrormakers.kafka.strimzi.io 0s55kafkas.kafka.strimzi.io 1s56kafkatopics.kafka.strimzi.io 1s57kafkausers.kafka.strimzi.io 1s58
59NOTES:60Thank you for installing strimzi-kafka-operator-0.14.061
62To create a Kafka cluster refer to the following documentation.63
64https://strimzi.io/docs/0.14.0/#kafka-cluster-str
List all the kubernetes objects created again:
1$ kubectl get all2NAME READY STATUS RESTARTS AGE3pod/strimzi-cluster-operator-6667fbc5f8-cqvdv 1/1 Running 0 9m25s4
5NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE6service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 90m7
8NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE9deployment.apps/strimzi-cluster-operator 1 1 1 1 9m25s10
11NAME DESIRED CURRENT READY AGE12replicaset.apps/strimzi-cluster-operator-6667fbc5f8 1 1 1 9m26s
We will now create a Kafka cluster with 3 brokers. The YAML file (kafka-cluster.Kafka.yaml
) for creating the Kafka cluster would like the following:
1apiVersion: kafka.strimzi.io/v1beta12kind: Kafka3metadata:4 name: kafka-cluster5spec:6 kafka:7 version: 2.3.0 # Kafka version8 replicas: 3 # Replicas specifies the number of broker nodes.9 listeners: # Listeners configure how clients connect to the Kafka cluster10 plain: {} # 909211 tls: {} # 909312 config:13 offsets.topic.replication.factor: 314 transaction.state.log.replication.factor: 315 transaction.state.log.min.isr: 216 log.message.format.version: "2.3"17 delete.topic.enable: "true"18 storage:19 type: persistent-claim20 size: 10Gi21 deleteClaim: false22 zookeeper:23 replicas: 324 storage:25 type: persistent-claim # Persistent storage backed by AWS EBS26 size: 10Gi27 deleteClaim: false28 entityOperator:29 topicOperator: {} # Operator for topic administration30 userOperator: {}
Apply the above YAML file:
1$ kubectl apply -f kafka-cluster.Kafka.yaml
This is where things get interesting. We will now analyse some of the k8s resources which the strimzi kafka operator
has created for us under the hood.
1$ kubectl get statefulsets.apps,pod,deployments,svc2NAME DESIRED CURRENT AGE3statefulset.apps/kafka-cluster-kafka 3 3 78m4statefulset.apps/kafka-cluster-zookeeper 3 3 79m5
6NAME READY STATUS RESTARTS AGE7pod/kafka-cluster-entity-operator-54cb77fd9d-9zbcx 3/3 Running 0 77m8pod/kafka-cluster-kafka-0 2/2 Running 0 78m9pod/kafka-cluster-kafka-1 2/2 Running 0 78m10pod/kafka-cluster-kafka-2 2/2 Running 0 78m11pod/kafka-cluster-zookeeper-0 2/2 Running 0 79m12pod/kafka-cluster-zookeeper-1 2/2 Running 0 79m13pod/kafka-cluster-zookeeper-2 2/2 Running 0 79m14pod/strimzi-cluster-operator-6667fbc5f8-cqvdv 1/1 Running 0 172m15
16NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE17deployment.extensions/kafka-cluster-entity-operator 1 1 1 1 77m18deployment.extensions/strimzi-cluster-operator 1 1 1 1 172m19
20NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE21service/kafka-cluster-kafka-bootstrap ClusterIP 10.100.177.177 <none> 9091/TCP,9092/TCP,9093/TCP 78m22service/kafka-cluster-kafka-brokers ClusterIP None <none> 9091/TCP,9092/TCP,9093/TCP 78m23service/kafka-cluster-zookeeper-client ClusterIP 10.100.199.128 <none> 2181/TCP 79m24service/kafka-cluster-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 79m25service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 4h13m
Points to note:
kafka-cluster-zookeeper
has created 3 pods - kafka-cluster-zookeeper-0
, kafka-cluster-zookeeper-1
and kafka-cluster-zookeeper-2
. The headless service kafka-cluster-zookeeper-nodes
facilitates network identity of these 3 pods (the 3 Zookeeper nodes).kafka-cluster-kafka
has created 3 pods - kafka-cluster-kafka-0
, kafka-cluster-kafka-1
and kafka-cluster-kafka-2
. The headless service kafka-cluster-kafka-brokers
facilitates network identity of these 3 pods (the 3 Kafka brokers).Persistent volumes are dynamically provisioned:
1$ kubectl get pv,pvc2NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE3persistentvolume/pvc-7ff2909f-e507-11e9-91df-0a1e73fdd786 10Gi RWO Delete Bound default/data-kafka-cluster-zookeeper-1 gp2 11h4persistentvolume/pvc-7ff290c4-e507-11e9-91df-0a1e73fdd786 10Gi RWO Delete Bound default/data-kafka-cluster-zookeeper-2 gp2 11h5persistentvolume/pvc-7ffd1d22-e507-11e9-a775-029ce0835b96 10Gi RWO Delete Bound default/data-kafka-cluster-zookeeper-0 gp2 11h6persistentvolume/pvc-a5997b77-e507-11e9-91df-0a1e73fdd786 10Gi RWO Delete Bound default/data-kafka-cluster-kafka-0 gp2 11h7persistentvolume/pvc-a599e52b-e507-11e9-91df-0a1e73fdd786 10Gi RWO Delete Bound default/data-kafka-cluster-kafka-1 gp2 11h8persistentvolume/pvc-a59c6cd2-e507-11e9-91df-0a1e73fdd786 10Gi RWO Delete Bound default/data-kafka-cluster-kafka-2 gp2 11h9
10NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE11persistentvolumeclaim/data-kafka-cluster-kafka-0 Bound pvc-a5997b77-e507-11e9-91df-0a1e73fdd786 10Gi RWO gp2 11h12persistentvolumeclaim/data-kafka-cluster-kafka-1 Bound pvc-a599e52b-e507-11e9-91df-0a1e73fdd786 10Gi RWO gp2 11h13persistentvolumeclaim/data-kafka-cluster-kafka-2 Bound pvc-a59c6cd2-e507-11e9-91df-0a1e73fdd786 10Gi RWO gp2 11h14persistentvolumeclaim/data-kafka-cluster-zookeeper-0 Bound pvc-7ffd1d22-e507-11e9-a775-029ce0835b96 10Gi RWO gp2 11h15persistentvolumeclaim/data-kafka-cluster-zookeeper-1 Bound pvc-7ff2909f-e507-11e9-91df-0a1e73fdd786 10Gi RWO gp2 11h16persistentvolumeclaim/data-kafka-cluster-zookeeper-2 Bound pvc-7ff290c4-e507-11e9-91df-0a1e73fdd786 10Gi RWO gp2 11h
You can view the provisioned AWS EBS volumes in the UI as well:
Before we get started with clients we need to create a topic (with 3 partitions and a replication factor of 3), over which our producer
and the consumer
and produce messages and consume messages on respectively.
1apiVersion: kafka.strimzi.io/v1beta12kind: KafkaTopic3metadata:4 name: test-topic5 labels:6 strimzi.io/cluster: kafka-cluster7spec:8 partitions: 39 replicas: 3
Apply the YAML to the k8s cluster:
1$ kubectl apply -f create-topics.yaml2kafkatopic.kafka.strimzi.io/test-topic created
The multi-broker Kafka cluster that we deployed is backed by statefulset
s and their corresponding headless service
s.
Since each Pod (Kafka broker) now has a network identity, clients can connect to the Kafka brokers via a combination of the pod name and service name: $(podname).$(governing service domain)
. In our case, these would be the following URLs:
kafka-cluster-kafka-0.kafka-cluster-kafka-brokers
kafka-cluster-kafka-1.kafka-cluster-kafka-brokers
kafka-cluster-kafka-2.kafka-cluster-kafka-brokers
Note:
$(podname).$(service name).$(namespace).svc.cluster.local
.kafka-cluster-kafka-bootstrap:9092
as well. It distributes the connection over the three broker specific endpoints I have listed above. As I no longer keep track of the individual broker endpoints, this method plays out well when I have to scale up or down the number of brokers in the Kafka cluster.First, clone this repo: {% github bensooraj/strimzi-kafka-aws-eks no-readme %}
1# Create the configmap, which contains details such as the broker DNS names, topic name and consumer group ID2$ kubectl apply -f test/k8s/config.yaml3configmap/kafka-client-config created4
5# Create the producer deployment6$ kubectl apply -f test/k8s/producer.Deployment.yaml7deployment.apps/node-test-producer created8
9# Expose the producer deployment via a service of type LoadBalancer (backed by the AWS Elastic Load Balancer). This just makes it easy for me to curl from postman10$ kubectl apply -f test/k8s/producer.Service.yaml11service/node-test-producer created12
13# Finally, create the consumer deployment14$ kubectl apply -f test/k8s/consumer.Deployment.yaml15deployment.apps/node-test-consumer created
If you list the producer service that we created, you would notice a URL
under EXTERNAL-IP:
1$ kubectl get svc2NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE3.4.5node-test-producer LoadBalancer 10.100.145.203 ac5f3d0d1e55a11e9a775029ce0835b9-2040242746.ap-south-1.elb.amazonaws.com 80:31231/TCP 55m
The URL ac5f3d0d1e55a11e9a775029ce0835b9-2040242746.ap-south-1.elb.amazonaws.com
is an AWS ELB
backed public endpoint which we will be querying for producing messages to the Kafka cluster.
Also, you can see that there is 1 producer and 3 consumers (one for each partition of the topic test-topic
):
1$ kubectl get pod2NAME READY STATUS RESTARTS AGE3node-test-consumer-96b44cbcb-gs2km 1/1 Running 0 125m4node-test-consumer-96b44cbcb-ptvjd 1/1 Running 0 125m5node-test-consumer-96b44cbcb-xk75j 1/1 Running 0 125m6node-test-producer-846d9c5986-vcsf2 1/1 Running 0 125m
The producer app basically exposes 3 URLs:
/kafka-test/green/:message
/kafka-test/blue/:message
/kafka-test/cyan/:message
Where :message
can be any valid string. Each of these URLs produce a message along with the colour information to the topic test-topic
.
The consumer group (the 3 consumer pods that we spin-up) listening for any incoming messages from the topic test-topic
, then receives these messages and prints them on to the console according to the colour instruction.
I curl
each URL 3 times. From the following GIF you can see how message consumption is distributed across the 3 consumers in a round-robin
manner:
1# Delete the test producer and consumer apps:2$ kubectl delete -f test/k8s/3configmap "kafka-client-config" deleted4deployment.apps "node-test-consumer" deleted5deployment.apps "node-test-producer" deleted6service "node-test-producer" deleted7
8# Delete the Kafka cluster9$ kubectl delete kafka kafka-cluster10kafka.kafka.strimzi.io "kafka-cluster" deleted11
12# Delete the Strimzi cluster operator13$ kubectl delete deployments. strimzi-cluster-operator14deployment.extensions "strimzi-cluster-operator" deleted15
16# Manually delete the persistent volumes17# Kafka18$ kubectl delete pvc data-kafka-cluster-kafka-019$ kubectl delete pvc data-kafka-cluster-kafka-120$ kubectl delete pvc data-kafka-cluster-kafka-221# Zookeeper22$ kubectl delete pvc data-kafka-cluster-zookeeper-023$ kubectl delete pvc data-kafka-cluster-zookeeper-124$ kubectl delete pvc data-kafka-cluster-zookeeper-2
Finally, delete the EKS cluster:
1$ eksctl delete cluster kafka-eks-cluster2[ℹ] using region ap-south-13[ℹ] deleting EKS cluster "kafka-eks-cluster"4[✔] kubeconfig has been updated5[ℹ] 2 sequential tasks: { delete nodegroup "ng-9f3cbfc7", delete cluster control plane "kafka-eks-cluster" [async] }6[ℹ] will delete stack "eksctl-kafka-eks-cluster-nodegroup-ng-9f3cbfc7"7[ℹ] waiting for stack "eksctl-kafka-eks-cluster-nodegroup-ng-9f3cbfc7" to get deleted8[ℹ] will delete stack "eksctl-kafka-eks-cluster-cluster"9[✔] all cluster resources were deleted
Hope this helped!
Note: This is not a tutorial per se, instead, this is me recording my observations as I setup a Kafka cluster for the first time on a Kubernetes platform using Strimzi.