The service mesh wars - a new hope for kubernetes

Watch talk on YouTube
  • Kuma
  • Linkerd
  • Cilium
  • Istio
  • Ambient Mesh

The new hope: Gateway API

  • Will integrate itself into the networking solution (nginx, istio, kong)
  • CRDs for Ingress, LB, Servicemesh
  • CRDs linke: Gateway, HttpRoute, GrpcRoute, TCPRoute

Expectations

  • Baseline: Control Plane and Data Plane (Application + Proxy)
  • What we get: Rules, Logs, …
  • Proxy-Variants:
    • Sidecar: Extra Pod, Service needs to be restarted for settings changes
    • Sidecarless: One proxy per node
  • Features: Ingress, egress, Mutual TLS, Retry Logic, Traffic Splitting, Ratelimits, Obervability

Comparison

Sidecar

TODO: Steal table from slides

| Kuma | Yes | Envoy |Linkerd | Yes | Linkerd Proxy

Features

TODO: Steal Diagrams from slides

  • Kuma: Gateway API Supported
    • CRD per Mesh with Ratelimiter, Timeouts, ….
    • To add to meh: Annotation
  • Linkerd: Gateway API Supported
    • Core Component: Server
    • To add to mesh: Annotate workload with proxy annotation
  • Cilium: Gateway API mostly Support
    • Utilizes eBPF for speed
    • Cann deploy envoy
    • CRDs for NEtworkPolicy
  • Istio: Gateway API Supported
    • CRDs with Services
    • To add: Annotate namespace or workload
  • Ambientmesh: Gateway API supported
    • Same Config as istio
    • Special: Layer 7 Rules require a waypoint
    • Missing: Several Policy features
    • To add: Annotate namespace and/or workload

TODO: Steal table from slides

Observability

  • Kuma: MEtrics by default with trace and log support (MeshTrace, MeshAccesslogs) via OpenTelemetry and it’s own UI
  • Linkerd: Prometheus metrics, Viz extension for UI and Jaeger extension for traces (not OTel compliant)
  • Cilium: No Traces, only metrics and logs through hubble (with ui)
  • Istio/Ambient: Metrics, Traces and Logs with full OTel support on Dataplane and a external UI (Kali)

TODO: Steal table

Performance

Tests: https://github.com/isItObservable/servicemeshsecuritybenchmark

  • KPIs: Ressources and Resource usage

  • Constant load, no policies:

    • Kuma 5,59ms
    • Linkerd: 2,55ms
    • Cilium 0ms
    • Istio: 6,43ms
    • Ambientmesh: 3,59ms

  • Loadtest no policies

    • Kuma: 7ms
    • Linkerd: 3,54ms
    • Cilium: 0,57ms
    • Istio: 8,8ms
    • Ambientmesh: 3,54ms

  • Constant load policies

    • Kuma: 6,08
    • Linkerd: 2,55
    • Cilium: 0
    • Istio: 9,19
    • Ambientmesh: 3,69

  • Loadtest: TODO

TODO: Steal overview slide

Recommendation

  • If ambientmesh supports everything you need: It performs the best
  • Kuma includes everything you need when starting your first mesh
  • Linkerd: Complex configuration
  • Treat cilium as your cni and not nessecarely as your servicemesh

TODO: Steal conclusion slide