Building air-gapped control planes for a global pharma leader using crossplane and argo

Joint effort of novo-nordik and upbound.

Background

• Ymir Platform: Foundational abstraction platform
• Goal: Faster time to market
• Usage in pharma: end-2-end compliance
• Airgap: Use gitopt and prevent human interaction with the control planes

Setup

• Decision for crossplane was obvious
• Problem: Chicken and egg “we provision clusters via crossplane but crossplane needs a cluster”
• GitOps: Everything as code with automatic tests and argo
• Infra: Azure

Public AKS

graph LR
    subgraph MC
        ProviderAzure
        ProviderKubernetes
    end
    ProviderAzure-->|Calls APU|AKS
    AKS-->|Provisions|Kubernetescluster
    ProviderKubernetes-->|Deploys service on|Kubernetescluster

Bastion Bootstrap

• Options: Terraform/Opentofu
• Goal: Crossplane all the things
• Solution: Run Crossplane in a github action
  1. Kind Cluster
  2. Install Crossplane
  3. Propagete Credentials
  4. Create Cluster
• Tooling: Uptest - E2E Test automation Framework, can be used for bootstrapping since it creates kind cluster with crossplane

graph LR
    subgraph GitHubRunner
        Kubernetes
        Crossplane
    end
    subgraph Azure
        BastionVM
    end
    Crossplane-->|Create|BastionVM

Next steps

• Problem: How to access bastion
• Solution: Auto-register bastion as github runner
• Create Bastion-Cluster via Uptest

graph LR
    subgraph Azure
        subgraph BastionVM
            GitHubRunner
            Kubernetes
            Crossplane
        end
        subgraph BastionCluster
            Kubernetes
            Argo
            CrossPlane
        end
    end
    Crossplane-->|Create|BastionCluster

TODO: Steal image from slides

Challenges

• Argo sync waves:
  • Problem: Argo does not support eventual consistency
  • Example: Install a ProviderConfig before your Provider and sync fails without retry
  • Order stuff very carefully
• Delivering updates to private clusters
  • Difference between public and private: It’s the same package
  • Upgrades/Downgrades: Change the package (Crossplane) and cluster (CRD)
• Testing:
  • Static: Multiple stages and each stage has it’s own bootstrap env that can be set to any branch
  • Ephemeral: Uptest

TODO: Steal images from slides

Wrap-up

• Cloud native air-gapped ✅
• GitOps ✅
• Crossplane, no terraform ✅
• Extensible, reusable, API-first ✅