— kubernetes, aws — 3 min read
When moving your services to the Kubernetes ecosystem for the first time, it is best practice to port only the stateless parts to begin with.
Here's the problem I had to solve: Our service uses Amazon RDS for MySQL. Both the RDS instance(s) and EKS reside within their own dedicated VPC. How do resources running within AWS EKS communicate with the database?
Let's dive right in!
We will be using the AWS CLI for setting up MySQL database.
We will first create a VPC with the CIDR block 10.0.0.0/24
which accommodate 254 hosts in all. This is more than enough to host our RDS instance.
1$ aws ec2 create-vpc --cidr-block 10.0.0.0/24 | jq '{VpcId:.Vpc.VpcId,CidrBlock:.Vpc.CidrBlock}'2{3 "VpcId": "vpc-0cf40a5f6db5eb3cd",4 "CidrBlock": "10.0.0.0/24"5}6
7# Export the RDS VPC ID for easy reference in the subsequent commands8$ export RDS_VPC_ID=vpc-0cf40a5f6db5eb3cd
RDS instances launched in a VPC must have a DB subnet group. DB subnet groups are a collection of subnets within a VPC. Each DB subnet group should have subnets
in at least two Availability Zones
in a given AWS Region
.
We will divide the RDS VPC (RDS_VPC_ID
) into two equal subnets: 10.0.0.0/25
and 10.0.0.128/25
.
So, let's create the first subnet in the availability zone ap-south-1b
:
1$ aws ec2 create-subnet --availability-zone "ap-south-1b" --vpc-id ${RDS_VPC_ID} --cidr-block 10.0.0.0/25 | jq '{SubnetId:.Subnet.SubnetId,AvailabilityZone:.Subnet.AvailabilityZone,CidrBlock:.Subnet.CidrBlock,VpcId:.Subnet.VpcId}'2# Response:3{4 "SubnetId": "subnet-042a4bee8e92287e8",5 "AvailabilityZone": "ap-south-1b",6 "CidrBlock": "10.0.0.0/25",7 "VpcId": "vpc-0cf40a5f6db5eb3cd"8}
and the second one in the availability zone ap-south-1a
1$ aws ec2 create-subnet --availability-zone "ap-south-1a" --vpc-id ${RDS_VPC_ID} --cidr-block 10.0.0.128/25 | jq '{SubnetId:.Subnet.SubnetId,AvailabilityZone:.Subnet.AvailabilityZone,CidrBlock:.Subnet.CidrBlock,VpcId:.Subnet.VpcId}'2# Response:3{4 "SubnetId": "subnet-0c01a5ba480b930f4",5 "AvailabilityZone": "ap-south-1a",6 "CidrBlock": "10.0.0.128/25",7 "VpcId": "vpc-0cf40a5f6db5eb3cd"8}
Each VPC has an implicit router which controls where network traffic is directed. Each subnet in a VPC must be explicitly associated with a route table, which controls the routing for the subnet.
Let's go ahead and associate these two subnet that we created, to the VPC's route table:
1# Fetch the route table information2$ aws ec2 describe-route-tables --filters Name=vpc-id,Values=${RDS_VPC_ID} | jq '.RouteTables[0].RouteTableId'3"rtb-0e680357de97595b1"4
5# For easy reference6$ export RDS_ROUTE_TABLE_ID=rtb-0e680357de97595b17
8# Associate the first subnet with the route table9$ aws ec2 associate-route-table --route-table-id rtb-0e680357de97595b1 --subnet-id subnet-042a4bee8e92287e810{11 "AssociationId": "rtbassoc-02198db22b2d36c97"12}13
14# Associate the second subnet with the route table15$ aws ec2 associate-route-table --route-table-id rtb-0e680357de97595b1 --subnet-id subnet-0c01a5ba480b930f416{17 "AssociationId": "rtbassoc-0e5c3959d360c92ab"18}
Now that we have two subnets spanning two availability zones, we can go ahead and create the DB subnet group.
1$ aws rds create-db-subnet-group --db-subnet-group-name "DemoDBSubnetGroup" --db-subnet-group-description "Demo DB Subnet Group" --subnet-ids "subnet-042a4bee8e92287e8" "subnet-0c01a5ba480b930f4" | jq '{DBSubnetGroupName:.DBSubnetGroup.DBSubnetGroupName,VpcId:.DBSubnetGroup.VpcId,Subnets:.DBSubnetGroup.Subnets[].SubnetIdentifier}'2# Response:3{4 "DBSubnetGroupName": "demodbsubnetgroup",5 "VpcId": "vpc-0cf40a5f6db5eb3cd",6 "Subnets": "subnet-0c01a5ba480b930f4"7}8{9 "DBSubnetGroupName": "demodbsubnetgroup",10 "VpcId": "vpc-0cf40a5f6db5eb3cd",11 "Subnets": "subnet-042a4bee8e92287e8"12}
The penultimate step to creating the DB instance is creating a VPC security group, an instance level virtual firewall with rules to control inbound and outbound traffic.
1$ aws ec2 create-security-group --group-name DemoRDSSecurityGroup --description "Demo RDS security group" --vpc-id ${RDS_VPC_ID}2{3 "GroupId": "sg-06800acf8d6279971"4}5
6# Export the RDS VPC Security Group ID for easy reference in the subsequent commands7$ export RDS_VPC_SECURITY_GROUP_ID=sg-06800acf8d6279971
We will use this security group at a later point, to set an inbound
rule to allow all traffic from the EKS cluster to the RDS instance.
1$ aws rds create-db-instance \2 --db-name demordsmyqldb \3 --db-instance-identifier demordsmyqldbinstance \4 --allocated-storage 10 \5 --db-instance-class db.t2.micro \6 --engine mysql \7 --engine-version "5.7.26" \8 --master-username demoappuser \9 --master-user-password demoappuserpassword \10 --no-publicly-accessible \11 --vpc-security-group-ids ${RDS_VPC_SECURITY_GROUP_ID} \12 --db-subnet-group-name "demodbsubnetgroup" \13 --availability-zone ap-south-1b \14 --port 3306 | jq '{DBInstanceIdentifier:.DBInstance.DBInstanceIdentifier,Engine:.DBInstance.Engine,DBName:.DBInstance.DBName,VpcSecurityGroups:.DBInstance.VpcSecurityGroups,EngineVersion:.DBInstance.EngineVersion,PubliclyAccessible:.DBInstance.PubliclyAccessible}'15
16# Respone:17{18 "DBInstanceIdentifier": "demordsmyqldbinstance",19 "Engine": "mysql",20 "DBName": "demordsmyqldb",21 "VpcSecurityGroups": [22 {23 "VpcSecurityGroupId": "sg-06800acf8d6279971",24 "Status": "active"25 }26 ],27 "EngineVersion": "5.7.26",28 "PubliclyAccessible": false29}
We can verify that the DB instance has been created in the UI as well:
Spinning up an EKS cluster on AWS is as simple as:
1$ eksctl create cluster --name=demo-eks-cluster --nodes=2 --region=ap-south-12[ℹ] using region ap-south-13[ℹ] setting availability zones to [ap-south-1a ap-south-1c ap-south-1b]4[ℹ] subnets for ap-south-1a - public:192.168.0.0/19 private:192.168.96.0/195[ℹ] subnets for ap-south-1c - public:192.168.32.0/19 private:192.168.128.0/196[ℹ] subnets for ap-south-1b - public:192.168.64.0/19 private:192.168.160.0/197[ℹ] nodegroup "ng-ae09882f" will use "ami-09c3eb35bb3be46a4" [AmazonLinux2/1.12]8[ℹ] creating EKS cluster "demo-eks-cluster" in "ap-south-1" region9[ℹ] will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup10[ℹ] if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-south-1 --name=demo-eks-cluster'11[ℹ] 2 sequential tasks: { create cluster control plane "demo-eks-cluster", create nodegroup "ng-ae09882f" }12[ℹ] building cluster stack "eksctl-demo-eks-cluster-cluster"13[ℹ] deploying stack "eksctl-demo-eks-cluster-cluster"14[ℹ] building nodegroup stack "eksctl-demo-eks-cluster-nodegroup-ng-ae09882f"15[ℹ] --nodes-min=2 was set automatically for nodegroup ng-ae09882f16[ℹ] --nodes-max=2 was set automatically for nodegroup ng-ae09882f17[ℹ] deploying stack "eksctl-demo-eks-cluster-nodegroup-ng-ae09882f"18[✔] all EKS cluster resource for "demo-eks-cluster" had been created19[✔] saved kubeconfig as "/Users/Bensooraj/.kube/config"20[ℹ] adding role "arn:aws:iam::account_number:role/eksctl-demo-eks-cluster-nodegroup-NodeInstanceRole-1631FNZJZTDSK" to auth ConfigMap21[ℹ] nodegroup "ng-ae09882f" has 0 node(s)22[ℹ] waiting for at least 2 node(s) to become ready in "ng-ae09882f"23[ℹ] nodegroup "ng-ae09882f" has 2 node(s)24[ℹ] node "ip-192-168-30-190.ap-south-1.compute.internal" is ready25[ℹ] node "ip-192-168-92-207.ap-south-1.compute.internal" is ready26[ℹ] kubectl command should work with "/Users/Bensooraj/.kube/config", try 'kubectl get nodes'27[✔] EKS cluster "demo-eks-cluster" in "ap-south-1" region is ready
We will create a kubernetes Service
named mysql-service
of type ExternalName
aliasing the RDS endpoint demordsmyqldbinstance.cimllxgykuy3.ap-south-1.rds.amazonaws.com
.
Run kubectl apply -f mysql-service.yaml
to create the service.
1# mysql-service.yaml2apiVersion: v13kind: Service4metadata:5 labels:6 app: mysql-service7 name: mysql-service8spec:9 externalName: demordsmyqldbinstance.cimllxgykuy3.ap-south-1.rds.amazonaws.com10 selector:11 app: mysql-service12 type: ExternalName13status:14 loadBalancer: {}
Now, clients running inside the pods within the cluster can connect to the RDS instance using mysql-service
.
Let's test the connect using a throwaway busybox
pod:
1$ kubectl run -i --tty --rm debug --image=busybox --restart=Never -- sh2If you don't see a command prompt, try pressing enter.3/ # nc mysql-service 33064^Cpunt!
It is evident that the pod is unable to get through! Let's solve the problem now.
We are going to create a VPC Peering Connection to facilitate communication between the resources in the two VPCs. According to the documentation:
A VPC peering connection is a networking connection between two VPCs that enables you to route traffic between them using private IPv4 addresses or IPv6 addresses. Instances in either VPC can communicate with each other as if they are within the same network. You can create a VPC peering connection between your own VPCs, or with a VPC in another AWS account. The VPCs can be in different regions (also known as an inter-region VPC peering connection).
To create a VPC peering connection, navigate to:
Peering Connections
and click on Create Peering Connection
.Requester
and the RDS VPC as the Accepter
):
Create Peering Connection
Peering Connection
that we just created. Click on Actions
=> Accept
. Again, in the confirmation dialog box, click on Yes, Accept
.
Don't forget to export the VPC Peering Connection ID:
1$ export VPC_PEERING_CONNECTION_ID=pcx-0cc408e65493fe197
1# Fetch the route table associated with the 3 public subnets of the VPC created by `eksctl`:2$ aws ec2 describe-route-tables --filters Name="tag:aws:cloudformation:logical-id",Values="PublicRouteTable" | jq '.RouteTables[0].RouteTableId'3"rtb-06103bd0704b3a9ee"4
5# For easy reference6export EKS_ROUTE_TABLE_ID=rtb-06103bd0704b3a9ee7
8# Add route: All traffic to (destination) the RDS VPC CIDR block is via the VPC Peering Connection (target)9$ aws ec2 create-route --route-table-id ${EKS_ROUTE_TABLE_ID} --destination-cidr-block 10.0.0.0/24 --vpc-peering-connection-id ${VPC_PEERING_CONNECTION_ID}10{11 "Return": true12}
1# Add route: All traffic to (destination) the EKS cluster CIDR block is via the VPC Peering Connection (target)2$ aws ec2 create-route --route-table-id ${RDS_ROUTE_TABLE_ID} --destination-cidr-block 192.168.0.0/16 --vpc-peering-connection-id ${VPC_PEERING_CONNECTION_ID}3{4 "Return": true5}
Allow all ingress traffic from the EKS cluster to the RDS instance on port 3306
:
1$ aws ec2 authorize-security-group-ingress --group-id ${RDS_VPC_SECURITY_GROUP_ID} --protocol tcp --port 3306 --cidr 192.168.0.0/16
1$ kubectl run -i --tty --rm debug --image=busybox --restart=Never -- sh2If you don't see a command prompt, try pressing enter.3/ # nc mysql-service 33064N55.7.26-logR&=lk`xTH???mj _5#K)>mysql_native_password
We can see that busybox
can now successfully talk to the RDS instance using the service mysql-service
.
That said, this is what our final setup looks like:
Note:
This setup allows all pods in the EKS cluster to access the RDS instance. Depending on your use case, this may or may not be ideal to your architecture. To implement more fine-grained access control, considering setting up a NetworkPolicy
resource.
Useful resources: