A note on dates: this post was written in 2020 using Terraform 0.12 and the nodejs12.x Lambda runtime. The architecture still holds up well, but if you’re following along today, check the current AWS provider docs and use a supported Lambda runtime and Terraform version. In particular, the aws_s3_bucket resource has since been split into smaller resources (aws_s3_bucket_acl, aws_s3_bucket_logging, aws_s3_bucket_website_configuration and so on).

Recently I had the chance to create a static website using S3 and CloudFront. I used Hugo as the framework for content management and Terraform to provision the infrastructure. There were a few interesting challenges, but in the end it worked out well.

In this talk, Chris Munns walks through building APIs with Amazon API Gateway - covering how it fits into a serverless architecture, the design choices you’ll face, and how to operate APIs in production. A solid primer if you’re putting an API Gateway in front of Lambda or your own backends.

In this talk, Paavan Mistry explains how to encrypt Kubernetes secrets at rest in etcd on Amazon EKS. By default, Kubernetes stores secrets only base64-encoded - not encrypted - so anyone who gains access to the API server or to etcd can read them in the clear. The talk shows how to use AWS KMS envelope encryption to protect secrets at rest, which is an easy win for anyone running EKS in production.

Bill Gates is famously a voracious reader, getting through dozens of books a year and publishing detailed recommendations on his blog, GatesNotes.

Watch a roundup of fifteen of his picks on YouTube →

What I like about his lists is how little of it is about technology. Gates reads widely - history, public health, psychology, science, biography - and that breadth clearly feeds into how he thinks about hard problems.

Amazon’s Leadership Principles are used in everything from hiring to design reviews, and they’ve become hugely influential well beyond Amazon itself.

Watch a walkthrough of the principles on YouTube →

The fourteen principles:

1. Customer Obsession
2. Ownership
3. Invent and Simplify
4. Are Right, A Lot
5. Learn and Be Curious
6. Hire and Develop the Best
7. Insist on the Highest Standards
8. Think Big
9. Bias for Action
10. Frugality
11. Earn Trust
12. Dive Deep
13. Have Backbone; Disagree and Commit
14. Deliver Results

(Amazon has since added two more - “Strive to be Earth’s Best Employer” and “Success and Scale Bring Broad Responsibility” - bringing the total to sixteen.)

Datadog can see almost everything in your platform. The problem is rarely missing data - it’s too much disconnected data, noisy alerts, and a bill that grows faster than your traffic. A few decisions made early make the difference between a tool your engineers trust and an expensive wall of dashboards.

This is the setup we recommend for running Datadog on Kubernetes.

Install the Agent the Kubernetes way

Deploy the Datadog Agent as a DaemonSet (one per node) together with the Cluster Agent, which collects cluster-level data and takes load off the Kubernetes API server. Either the Datadog Operator or the official Helm chart will do this; pick one and keep its configuration in Git.

Amazon’s Leadership Principles are used in everything from hiring to design reviews, and they’ve become hugely influential well beyond Amazon itself.

Watch a walkthrough of the principles on YouTube →

The fourteen principles:

1. Customer Obsession
2. Ownership
3. Invent and Simplify
4. Are Right, A Lot
5. Learn and Be Curious
6. Hire and Develop the Best
7. Insist on the Highest Standards
8. Think Big
9. Bias for Action
10. Frugality
11. Earn Trust
12. Dive Deep
13. Have Backbone; Disagree and Commit
14. Deliver Results

(Amazon has since added two more - “Strive to be Earth’s Best Employer” and “Success and Scale Bring Broad Responsibility” - bringing the total to sixteen.)

Arnold Schwarzenegger gave this commencement speech at the University of Southern California, and the six rules he lays out have stuck with me. They’re simple, but they apply just as much to a career in engineering as to anything else.

Watch Arnold’s full USC commencement speech on YouTube →

His six rules:

1. Trust yourself. Work out who you really are and what you want, rather than what others expect of you.
2. Break the rules. Not the laws - the conventions. You rarely do anything remarkable by following the well-worn path.
3. Don’t be afraid to fail. You can’t win without being willing to lose. Failure is the cost of trying anything worthwhile.
4. Don’t listen to the naysayers. “It can’t be done” usually means “I couldn’t do it.”
5. Work your butt off. There is no substitute for the hours. Talent without effort gets overtaken.
6. Give back. Reaching the top means nothing if you don’t reach back and help others up.

What I take from it for our work: most of the interesting problems in cloud and reliability engineering come from being willing to question how things are “normally” done, putting in the unglamorous hours to get the details right, and then sharing what you learn. That’s a big part of why this blog exists.

Bill Gates is famously a voracious reader, getting through dozens of books a year and publishing detailed recommendations on his blog, GatesNotes.

Watch a roundup of fifteen of his picks on YouTube →

What I like about his lists is how little of it is about technology. Gates reads widely - history, public health, psychology, science, biography - and that breadth clearly feeds into how he thinks about hard problems.

Have you ever wondered how your browser resolves a domain name when you hit Enter on a URL? Here’s how it works, at a high level.

1. Your browser first looks in its own cache for the address. If it doesn’t find it, it checks the OS cache, and finally the ISP’s cache.

2. If the name isn’t cached, the next step is the /etc/hosts file. If there’s an IP address there, the browser tries to load the page from that server.

A service mesh moves the hard parts of service-to-service networking - encryption, retries, timeouts, traffic shifting and telemetry - out of your application code and into the platform. Istio is the most widely used mesh on Kubernetes, and when it’s adopted deliberately it removes a lot of bespoke plumbing. Adopted all at once, it can also add a layer of complexity your team isn’t ready to debug.

This post covers what Istio gives you, the features worth turning on first, and the gotchas we’ve hit running it in production.

This talk by Dawid Ziolkowski at DevOpsDays Warsaw is a clear introduction to Istio. If you’ve heard of service meshes but never quite grasped what problems they solve, it’s a good place to start - it covers what a mesh gives you (traffic management, observability and mutual TLS between services) and how Istio implements it on top of Kubernetes.

By default, the JVM’s maximum heap size is 1/4 of the physical memory available - you can read about this in the Oracle GC tuning guide. This means that if you don’t define -Xmx in your JVM parameters, the container will set 1/4 of the host memory as the maximum heap size.

On a recent enough JVM (8u191+ and 10+), the JVM is container-aware: if you set a Kubernetes resource limit, the JVM uses that limit rather than the host’s memory to size the heap. So your maximum heap size becomes 1/4 of your Kubernetes resource limit.

A service mesh moves the hard parts of service-to-service networking - encryption, retries, timeouts, traffic shifting and telemetry - out of your application code and into the platform. Istio is the most widely used mesh on Kubernetes, and when it’s adopted deliberately it removes a lot of bespoke plumbing. Adopted all at once, it can also add a layer of complexity your team isn’t ready to debug.

This post covers what Istio gives you, the features worth turning on first, and the gotchas we’ve hit running it in production.

Datadog can see almost everything in your platform. The problem is rarely missing data - it’s too much disconnected data, noisy alerts, and a bill that grows faster than your traffic. A few decisions made early make the difference between a tool your engineers trust and an expensive wall of dashboards.

This is the setup we recommend for running Datadog on Kubernetes.

Install the Agent the Kubernetes way

Deploy the Datadog Agent as a DaemonSet (one per node) together with the Cluster Agent, which collects cluster-level data and takes load off the Kubernetes API server. Either the Datadog Operator or the official Helm chart will do this; pick one and keep its configuration in Git.

By default, the JVM’s maximum heap size is 1/4 of the physical memory available - you can read about this in the Oracle GC tuning guide. This means that if you don’t define -Xmx in your JVM parameters, the container will set 1/4 of the host memory as the maximum heap size.

On a recent enough JVM (8u191+ and 10+), the JVM is container-aware: if you set a Kubernetes resource limit, the JVM uses that limit rather than the host’s memory to size the heap. So your maximum heap size becomes 1/4 of your Kubernetes resource limit.

A Service Account provides an identity for processes that run in a pod. When processes inside a pod contact the API server, they are authenticated as a particular Service Account.

Create a Service Account using the YAML below.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-app-name
  namespace: namespace-name

Once the Service Account is created, you can reference it in your pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: app-name
spec:
  serviceAccountName: sa-app-name

Now on to Kubernetes Secrets. A Secret lets you store and manage sensitive information. Here we’ll create two secrets for our tests: mysecret1 and mysecret2. Note the --- separator - without it, the second document silently overwrites the first.

This talk by Dawid Ziolkowski at DevOpsDays Warsaw is a clear introduction to Istio. If you’ve heard of service meshes but never quite grasped what problems they solve, it’s a good place to start - it covers what a mesh gives you (traffic management, observability and mutual TLS between services) and how Istio implements it on top of Kubernetes.

In this talk, Paavan Mistry explains how to encrypt Kubernetes secrets at rest in etcd on Amazon EKS. By default, Kubernetes stores secrets only base64-encoded - not encrypted - so anyone who gains access to the API server or to etcd can read them in the clear. The talk shows how to use AWS KMS envelope encryption to protect secrets at rest, which is an easy win for anyone running EKS in production.

Have you ever wondered how your browser resolves a domain name when you hit Enter on a URL? Here’s how it works, at a high level.

1. Your browser first looks in its own cache for the address. If it doesn’t find it, it checks the OS cache, and finally the ISP’s cache.

2. If the name isn’t cached, the next step is the /etc/hosts file. If there’s an IP address there, the browser tries to load the page from that server.

Datadog can see almost everything in your platform. The problem is rarely missing data - it’s too much disconnected data, noisy alerts, and a bill that grows faster than your traffic. A few decisions made early make the difference between a tool your engineers trust and an expensive wall of dashboards.

This is the setup we recommend for running Datadog on Kubernetes.

Install the Agent the Kubernetes way

Deploy the Datadog Agent as a DaemonSet (one per node) together with the Cluster Agent, which collects cluster-level data and takes load off the Kubernetes API server. Either the Datadog Operator or the official Helm chart will do this; pick one and keep its configuration in Git.

A service mesh moves the hard parts of service-to-service networking - encryption, retries, timeouts, traffic shifting and telemetry - out of your application code and into the platform. Istio is the most widely used mesh on Kubernetes, and when it’s adopted deliberately it removes a lot of bespoke plumbing. Adopted all at once, it can also add a layer of complexity your team isn’t ready to debug.

This post covers what Istio gives you, the features worth turning on first, and the gotchas we’ve hit running it in production.

A Service Account provides an identity for processes that run in a pod. When processes inside a pod contact the API server, they are authenticated as a particular Service Account.

Create a Service Account using the YAML below.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-app-name
  namespace: namespace-name

Once the Service Account is created, you can reference it in your pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: app-name
spec:
  serviceAccountName: sa-app-name

Now on to Kubernetes Secrets. A Secret lets you store and manage sensitive information. Here we’ll create two secrets for our tests: mysecret1 and mysecret2. Note the --- separator - without it, the second document silently overwrites the first.

In this talk, Paavan Mistry explains how to encrypt Kubernetes secrets at rest in etcd on Amazon EKS. By default, Kubernetes stores secrets only base64-encoded - not encrypted - so anyone who gains access to the API server or to etcd can read them in the clear. The talk shows how to use AWS KMS envelope encryption to protect secrets at rest, which is an easy win for anyone running EKS in production.

In this talk, Chris Munns walks through building APIs with Amazon API Gateway - covering how it fits into a serverless architecture, the design choices you’ll face, and how to operate APIs in production. A solid primer if you’re putting an API Gateway in front of Lambda or your own backends.

A service mesh moves the hard parts of service-to-service networking - encryption, retries, timeouts, traffic shifting and telemetry - out of your application code and into the platform. Istio is the most widely used mesh on Kubernetes, and when it’s adopted deliberately it removes a lot of bespoke plumbing. Adopted all at once, it can also add a layer of complexity your team isn’t ready to debug.

This post covers what Istio gives you, the features worth turning on first, and the gotchas we’ve hit running it in production.

This talk by Dawid Ziolkowski at DevOpsDays Warsaw is a clear introduction to Istio. If you’ve heard of service meshes but never quite grasped what problems they solve, it’s a good place to start - it covers what a mesh gives you (traffic management, observability and mutual TLS between services) and how Istio implements it on top of Kubernetes.

Datadog can see almost everything in your platform. The problem is rarely missing data - it’s too much disconnected data, noisy alerts, and a bill that grows faster than your traffic. A few decisions made early make the difference between a tool your engineers trust and an expensive wall of dashboards.

This is the setup we recommend for running Datadog on Kubernetes.

Install the Agent the Kubernetes way

Deploy the Datadog Agent as a DaemonSet (one per node) together with the Cluster Agent, which collects cluster-level data and takes load off the Kubernetes API server. Either the Datadog Operator or the official Helm chart will do this; pick one and keep its configuration in Git.

A note on dates: this post was written in 2020 using Terraform 0.12 and the nodejs12.x Lambda runtime. The architecture still holds up well, but if you’re following along today, check the current AWS provider docs and use a supported Lambda runtime and Terraform version. In particular, the aws_s3_bucket resource has since been split into smaller resources (aws_s3_bucket_acl, aws_s3_bucket_logging, aws_s3_bucket_website_configuration and so on).

Recently I had the chance to create a static website using S3 and CloudFront. I used Hugo as the framework for content management and Terraform to provision the infrastructure. There were a few interesting challenges, but in the end it worked out well.