Skip to content
Ben Sooraj

Sending application logs from EKS to Datadog

architecture2 min read

Datadog has been part of our stack for quite sometime now, where the datadog-agent installed on the EC2 instances (baked into the image) tail log files to which the application services write to.

A recent project required me to send the application logs deployed on kubernetes to Datadog. This required a different approach which I would be sharing today. A few pre-requisites before I begin:

  1. An up and running kubernetes cluster. I use eksctl to quickly spin up a k8s cluster on AWS EKS.
  2. A Datadog account
  3. Helm v3 installed (v3 doesn't require you to setup Tiller (Helm's server component) on your kubernetes cluster)

Pre-requisites

To set the context, I have a 3-node cluster set up:

1$ kubectl get nodes
2NAME STATUS ROLES AGE VERSION
3ip-10-0-0-34.ap-south-1.compute.internal Ready <none> 9d v1.16.13-eks-2ba888
4ip-10-0-1-144.ap-south-1.compute.internal Ready <none> 9d v1.16.13-eks-2ba888
5ip-10-0-2-79.ap-south-1.compute.internal Ready <none> 9d v1.16.13-eks-2ba888

Add and update the Datadog chart repository,

1# Ensure that you are running helm v3
2$ helm version --short
3v3.3.1+g249e521
4
5# Add and update the Datadog and stable chart repositories
6$ helm repo add datadog https://helm.datadoghq.com
7"datadog" has been added to your repositories
8
9$ helm repo add stable https://kubernetes-charts.storage.googleapis.com/
10"stable" has been added to your repositories
11
12$ helm repo update
13Hang tight while we grab the latest from your chart repositories...
14...Successfully got an update from the "stable" chart repository
15Update Complete. ⎈Happy Helming!⎈
16
17$ helm repo list
18NAME URL
19stable https://kubernetes-charts.storage.googleapis.com/
20datadog https://helm.datadoghq.com
21
22# Search for datadog chart. We will be using datadog/datadog
23$ helm search repo datadog
24NAME CHART VERSION APP VERSION DESCRIPTION
25datadog/datadog 2.4.13 7 Datadog Agent
26stable/datadog 2.3.42 7 DEPRECATED Datadog Agent

Clone the git repository bensooraj/node-eks-datadog-integration:

1$ git clone https://github.com/bensooraj/node-eks-datadog-integration.git
2
3# A quick overview overview of the project structure
4$ tree -L 2
5.
6├── Makefile
7├── README.md
8├── coverage
9├── datadog
10│ └── values.yml
11├── docker
12│ └── Dockerfile
13├── kubernetes
14│ ├── deployment.yaml
15│ └── service.yaml
16├── node_modules
17│ ├── @dabh
18│ ....
19│ ...
20│ ..
21│ ├── winston
22│ └── winston-transport
23├── package-lock.json
24├── package.json
25└── server.js
26
2783 directories, 9 files

Check datadog-values.yaml for the latest YAML file.

Deploy the application

We will be deploying an ultra simple Node.js API which logs out JSON formatted log messages whenever it receives a GET request:

1// server.js
2const express = require('express');
3const winston = require('winston');
4
5// Constants
6const PORT = 8080;
7const HOST = '0.0.0.0';
8
9const app = express();
10
11const logger = winston.createLogger({
12 transports: [
13 new winston.transports.Console({
14 handleExceptions: true,
15 })
16 ],
17 level: 'debug',
18 exitOnError: false,
19 format: winston.format.combine(
20 winston.format.splat(),
21 winston.format.timestamp(),
22 winston.format.json()
23 ),
24});
25
26app.get('/', (req, res) => {
27 logger.silly("This is a silly message", { url: req.url, environment: "test" });
28 logger.debug("This is a debug message", { url: req.url, environment: "test" });
29 logger.info("This is an info message", { url: req.url, environment: "test" });
30
31 res.json({
32 response_message: 'Hello World'
33 });
34});
35
36app.listen(PORT, HOST);
37logger.info("Running on http://${HOST}:${PORT}", { host: HOST, port: PORT, environment: "test" });

Relevant sections from the deployment YAML file,

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: eks-datadog-demo-node-app
5 annotations:
6 ad.datadoghq.com/eks-datadog-demo-node-app.logs: '[{"source":"nodejs","service":"eks-datadog-demo-node-app"}]'
7 labels:
8 tags.datadoghq.com/env: "test"
9 tags.datadoghq.com/service: "eks-datadog-demo-node-app"
10 tags.datadoghq.com/version: "v1.0.1"
11spec:
12 # # #
13 # #
14 #
15 template:
16 metadata:
17 annotations:
18 ad.datadoghq.com/eks-datadog-demo-node-app.logs: '[{"source":"nodejs","service":"eks-datadog-demo-node-app"}]'
19 labels:
20 app: eks-datadog-demo-node-app
21 tags.datadoghq.com/env: "test"
22 tags.datadoghq.com/service: "eks-datadog-demo-node-app"
23 tags.datadoghq.com/version: "v1.0.1"
24 spec:
25 containers:
26 - name: eks-datadog-demo-node-app
27 # # #
28 # #
29 #

Pay close attention to the labels and annotations. These are especially useful if you are planning to collect traces and application metrics, which are outside the scope of this article.

It's worth mentioning that you can exclude or include logs from deployments and/or pods by configuring the settings in the annotations. For example, to exclude all logs from the above deployment configure the log_processing_rules in the annotations section as follows:

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: eks-datadog-demo-node-app
5 annotations:
6 ad.datadoghq.com/eks-datadog-demo-node-app.logs: >-
7 [{
8 "source": "nodejs",
9 "service": "eks-datadog-demo-node-app",
10 "log_processing_rules": [{
11 "type": "exclude_at_match",
12 "name": "exclude_this_deployment",
13 "pattern":"\.*"
14 }]
15 }]
16 labels:
17 tags.datadoghq.com/env: "test"
18 tags.datadoghq.com/service: "eks-datadog-demo-node-app"
19 tags.datadoghq.com/version: "v1.0.1"
20spec:
21 # # #
22 # #
23 #

Deploy the API and exopse it as a service,

1# Deployment
2$ kubectl apply -f kubernetes/deployment.yaml
3
4# Service
5$ kubectl apply -f kubernetes/service.yaml
6
7# Verify that the API is up and running
8$ kubectl get deployment,svc,po
9NAME READY UP-TO-DATE AVAILABLE AGE
10deployment.apps/eks-datadog-demo-node-app 2/2 2 2 30h
11
12NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
13service/eks-datadog-demo-node-app ClusterIP 172.20.205.150 <none> 8080/TCP 30h
14
15NAME READY STATUS RESTARTS AGE
16pod/eks-datadog-demo-node-app-7bf7cf764f-gpqvf 1/1 Running 0 29h
17pod/eks-datadog-demo-node-app-7bf7cf764f-spb7m 1/1 Running 0 29h

Deploy the Datadog agent

Open the file datadog/values.yml and ensure that datadog.logs.enabled and datadog.logs.containerCollectAll are set to true.

1datadog:
2 ## @param logs - object - required
3 ## Enable logs agent and provide custom configs
4 #
5 logs:
6 ## @param enabled - boolean - optional - default: false
7 ## Enables this to activate Datadog Agent log collection.
8 #
9 enabled: true
10 ## @param containerCollectAll - boolean - optional - default: false
11 ## Enable this to allow log collection for all containers.
12 #
13 containerCollectAll: true

Deploy the datadog-agent:

1$ helm install datadog-agent -f datadog/values.yml --set datadog.apiKey=<DATADOG_API_KEY> datadog/datadog
2
3# Helm will deploy one datadog-agent pod per node
4$ kubectl get deployment,svc,po
5NAME READY UP-TO-DATE AVAILABLE AGE
6deployment.apps/datadog-agent-kube-state-metrics 1/1 1 1 29h
7
8NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
9service/datadog-agent-kube-state-metrics ClusterIP 172.20.129.97 <none> 8080/TCP 29h
10
11NAME READY STATUS RESTARTS AGE
12pod/datadog-agent-b4mtv 2/2 Running 0 29h
13pod/datadog-agent-f6jn7 2/2 Running 0 29h
14pod/datadog-agent-zvp6p 2/2 Running 0 29h

Note that even though the agent is deployed in the default namespace, it can collect logs and metrics from deployments/pods from all namespaces.

O Logs! Where art thou?

Now we have the API service (for generating logs) and the Datadog agent (for streaming the logs to Datadog) all set! I assume this is what happens behind the scenes:

  1. The API service writes logs to stdout and stderr
  2. The kubelet writes these logs to /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/<num>.log on the node machine
  3. The datadog-agents running on each node tails these log files and streams them to the Datadog servers

Anyways, let's generate some logs:

1# Port forward to access the API service from your local machine
2$ kubectl port-forward service/eks-datadog-demo-node-app 8080:8080
3Forwarding from 127.0.0.1:8080 -> 8080
4Forwarding from [::1]:8080 -> 8080
5
6# Generate some logs using curl or ab
7$ ab -n 50 -c 2 -l http://localhost:8080/
8
9# Verify that the logs are generated
10j$ kubectl logs -l app=eks-datadog-demo-node-app --since=100m
11{"url":"/","environment":"test","level":"debug","message":"This is a debug message","timestamp":"2020-09-04T19:35:52.126Z"}
12{"url":"/","environment":"test","level":"info","message":"This is an info message","timestamp":"2020-09-04T19:35:53.308Z"}
13....
14...
15{"url":"/","environment":"test","level":"debug","message":"This is a debug message","timestamp":"2020-09-04T19:35:53.310Z"}
16{"url":"/","environment":"test","level":"info","message":"This is an info message","timestamp":"2020-09-04T19:35:53.310Z"}

Navigate to Log Explorer in Datadog to see the logs that we generated above

Datadog UI overview

Click on any one of the log records to view more details

Datadog log record detail view

On the left pane, you can filter the logs by namespace, pods etc. as well!

Datadog log record detail view UI

That's it for now!

I hope this helped and if you have any feedback for me, please let me know :smile:.

Further reading:

  1. Logging Architecture
  2. Datadog kubernetes log collection
  3. Datadog agent as a DaemonSet (recommended)

Note: This article is not an in-depth tutorial on implemending logging for applications running on kubernetes, but a journal of my learnings.