A service mesh moves the hard parts of service-to-service networking - encryption, retries, timeouts, traffic shifting and telemetry - out of your application code and into the platform. Istio is the most widely used mesh on Kubernetes, and when it’s adopted deliberately it removes a lot of bespoke plumbing. Adopted all at once, it can also add a layer of complexity your team isn’t ready to debug.

This post covers what Istio gives you, the features worth turning on first, and the gotchas we’ve hit running it in production.

Datadog can see almost everything in your platform. The problem is rarely missing data - it’s too much disconnected data, noisy alerts, and a bill that grows faster than your traffic. A few decisions made early make the difference between a tool your engineers trust and an expensive wall of dashboards.

This is the setup we recommend for running Datadog on Kubernetes.

Install the Agent the Kubernetes way

Deploy the Datadog Agent as a DaemonSet (one per node) together with the Cluster Agent, which collects cluster-level data and takes load off the Kubernetes API server. Either the Datadog Operator or the official Helm chart will do this; pick one and keep its configuration in Git.

By default, the JVM’s maximum heap size is 1/4 of the physical memory available - you can read about this in the Oracle GC tuning guide. This means that if you don’t define -Xmx in your JVM parameters, the container will set 1/4 of the host memory as the maximum heap size.

On a recent enough JVM (8u191+ and 10+), the JVM is container-aware: if you set a Kubernetes resource limit, the JVM uses that limit rather than the host’s memory to size the heap. So your maximum heap size becomes 1/4 of your Kubernetes resource limit.

A Service Account provides an identity for processes that run in a pod. When processes inside a pod contact the API server, they are authenticated as a particular Service Account.

Create a Service Account using the YAML below.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-app-name
  namespace: namespace-name

Once the Service Account is created, you can reference it in your pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: app-name
spec:
  serviceAccountName: sa-app-name

Now on to Kubernetes Secrets. A Secret lets you store and manage sensitive information. Here we’ll create two secrets for our tests: mysecret1 and mysecret2. Note the --- separator - without it, the second document silently overwrites the first.

This talk by Dawid Ziolkowski at DevOpsDays Warsaw is a clear introduction to Istio. If you’ve heard of service meshes but never quite grasped what problems they solve, it’s a good place to start - it covers what a mesh gives you (traffic management, observability and mutual TLS between services) and how Istio implements it on top of Kubernetes.

In this talk, Paavan Mistry explains how to encrypt Kubernetes secrets at rest in etcd on Amazon EKS. By default, Kubernetes stores secrets only base64-encoded - not encrypted - so anyone who gains access to the API server or to etcd can read them in the clear. The talk shows how to use AWS KMS envelope encryption to protect secrets at rest, which is an easy win for anyone running EKS in production.