Cloudy with a chance of kubernetes

Watch talk on YouTube Slides

Background

  • Scale: 100s of clusters
  • Cloud: Azure, AWS, GCP
  • The baseline: Single AWS Region and applications on vms
  • Goal: Operate on different locations
  • History: They added more and more regions - 6 Providers in 6 Regions across 29 locations
  • Problem: Different tooling across different cloud providers
  • Idea: Kubernetes abstracts the specific cloud provider infra

The way

  • Idea: Use managed kubernetes
  • Problem: In 2018 the managed offerings were in beta or very limited
  • Challenge: Opinionated cloud specific stuff

Iterations

  1. Clusters based on vms created by terraform and other automation tools -> They realized that they need multiple clusters per region
  2. Their own application delivery platform that deployed to the right clusters across regions for better DevEx
  3. k8s on k8s (hosted cp) -> Current setup with a terraform managed parent cluster
  4. Idea: Host the Partent-Cluster on managed kubernetes -> They need to abstract some things away
  5. Solution: Use their good old aplication delivery platform

Abstractions

  • Use custom CRDs to abstract the same behaviour across providers