CI: full CAPI-provider-stack e2e validating ALL features against a real backend (iteration 2)

Why

Iteration 1 (#17 (closed)) validates the getting-started happy path end-to-end on a real backend — base cluster + app + the optional storage and LoadBalancer steps — and builds the reusable harness. This issue is iteration 2: extend that harness to the full feature matrix — every advanced feature beyond the getting-started walkthrough — so a regression in any of them is caught, not just the documented happy path.

Scope (iteration 2 — real provisioning, all features)

Build on the iteration-1 harness (kind mgmt cluster + built provider + real credentials/location from #17 (closed)), and drive each advanced feature — the ones not covered by the getting-started walkthrough — to a healthy cluster + teardown:

  • (base cluster + app + basic PVC bind + basic Service-LB are #17 (closed)'s getting-started happy path — not repeated here; #18 (closed) starts at the advanced features below.)
  • prebaked — boots a pre-baked image; the CP-upgrade local-apiserver fix present.
  • HA control plane — 3 CP nodes, etcd quorum, a safe rolling upgrade.
  • geo (single-location) and geo-multilocation — cloudspace mesh (MeshReady), health-aware DNS endpoint (EndpointReady), calico-geo MTU, apiserver-proxy, failover behaviour.
  • CCM — nodes de-tainted, Service type=LoadBalancer gets an external IP.
  • backup — scheduled etcd backup to S3 (Object Lock), restore smoke.
  • air-gapped — in-cloudspace registry, zero-egress node boot.
  • storage/CSI — PVC binds to a real disk, online expansion, (encrypted class if a vault is wired).
  • disks / networking — physical etcd disk, pod connectivity across sites.
  • Kubernetes upgrade — new image + rolling replace across a minor.
  • teardown — clean release of all whitesky.cloud resources (no leaks).

Each maps to an assertion set mirroring the guide it documents.

Constraints / design

  • Real credentials in CI (masked/protected variables), scoped to a dedicated test customer + location; per-cluster credential model per §12/§18.
  • Cost + time: provisions real VMs and multi-cloudspace geo — too slow/expensive for every MR. Run nightly / manual / pre-release (scheduled pipeline or a manual gate), not on merge_request_event.
  • Isolation + cleanup: unique names per run; guaranteed teardown (even on failure) to avoid orphaned cloudspaces/VMs/disks and cost leaks. A reaper for stragglers.
  • Likely a matrix or staged suite; heavy features (geo, air-gapped) may be opt-in tiers.

Acceptance

A scheduled/manual pipeline that stands up each feature against the real backend, asserts the documented behaviour, and tears everything down cleanly — surfacing any feature regression before a release. Depends on and reuses the harness from #17 (closed).

Edited by Geert Audenaert