Cloudy with a chance of kubernetes
Slides
Background
- Scale: 100s of clusters
- Cloud: Azure, AWS, GCP
- The baseline: Single AWS Region and applications on vms
- Goal: Operate on different locations
- History: They added more and more regions - 6 Providers in 6 Regions across 29 locations
- Problem: Different tooling across different cloud providers
- Idea: Kubernetes abstracts the specific cloud provider infra
The way
- Idea: Use managed kubernetes
- Problem: In 2018 the managed offerings were in beta or very limited
- Challenge: Opinionated cloud specific stuff
Iterations
- Clusters based on vms created by terraform and other automation tools -> They realized that they need multiple clusters per region
- Their own application delivery platform that deployed to the right clusters across regions for better DevEx
- k8s on k8s (hosted cp) -> Current setup with a terraform managed parent cluster
- Idea: Host the Partent-Cluster on managed kubernetes -> They need to abstract some things away
- Solution: Use their good old aplication delivery platform
Abstractions
- Use custom CRDs to abstract the same behaviour across providers