Getting Started: CNI never lands on the from-release path — ship addon CRS in releases + one-time install step (kill the NotReady trap)

Symptom

A new user follows the docs exactly (install-provider.md → first-cluster.md), applies a generated cluster, and the control-plane node sits NotReady forever:

Node.Ready: container runtime network not ready: NetworkPluginNotReady
  cni plugin not initialized

It looks like a failed deploy. The natural reaction is "delete and retry" — which changes nothing, because the cause is structural. This is the single worst first-run experience we have and it will hit 100% of new users.

Root cause — it's not that the mechanism is missing

The CNI-via-ClusterResourceSet machinery is already built and correct:

  • config/addons/cni/ — CRS capvco-cni-calico (Reconcile), selector capvco.infrastructure.cluster.x-k8s.io/cni: calico, Calico pinned to POD_CIDR default 10.244.0.0/16.
  • config/addons/cni-geo/ — CRS capvco-cni-calico-geo, veth_mtu: "1350" for the connected-cloudspace S2S tunnel (the MTU research from 2026-07-10).
  • The cni: labels are correctly on the Cluster object (what CRS matches). Same pattern for csi/, ccm/, service-lb/, etc.

The gap is delivery, two failures stacked:

  1. Addons aren't shipped in releases. A release ships the provider bundle + templates + CLIs, but not the config/addons/** CRS bundles. A clusterctl init-from-registry user has no config/addons/ to apply.
  2. The from-release docs never tell them to apply it. The CRS install is only documented in access-and-deploy.md (the contributor out-of-cluster flow). install-provider.md / first-cluster.md say nothing — so the templates stamp cni: calico, but nothing on the management cluster is listening for that label. The label points at a mechanism that was never installed.

(Verified live: a from-release cluster stuck at NotReady; applying the bundled config/addons/cni Calico by hand → node Ready in ~10s, coredns pod on 10.244.215.x. Pure delivery gap, not a code bug.)

Fix — "Deploy your first cluster" must be smooth end-to-end, nothing can go wrong

1. Ship the addon CRS bundles in the release

Publish self-contained, kustomize-built bundles to the package registry alongside the templates (same out/ → packages/generic/capvco/vX.Y.Z/ path), so no repo checkout is ever needed:

  • cni.yaml (ConfigMap + CRS from config/addons/cni) — the first-cluster blocker
  • cni-geo.yaml (from config/addons/cni-geo) — geo users
  • csi.yaml, service-lb.yaml — the base template also labels csi: vco; without CSI, PVCs never bind (smooth ≠ just Ready). Ship these too (or clearly scope them as opt-in in the guides).

make release-manifests should build these from config/addons/** so the release and the repo can't drift.

2. Add ONE install step to install-provider.md

Right after clusterctl init, apply the CNI CRS bundle onto the management cluster, from the release URL (never a local path):

kubectl apply -f https://gitlab.whitesky.cloud/api/v4/projects/730/packages/generic/capvco/<version>/cni.yaml
# geo clusters also: .../cni-geo.yaml

One apply, once per management cluster. Every cluster then self-installs its CNI.

3. Make first-cluster.md reassuring and self-verifying

  • State plainly that the CNI is delivered automatically by the CRS once the API server is up, and that a brief NotReady → Ready transition (tens of seconds) is expected — so nobody mistakes it for a failure.
  • Add an explicit checkpoint: kubectl get nodes shows Ready; kubectl get pods -A shows calico-node Running. Show the expected output.

4. Landmines to close (this is the "nothing can go wrong" part)

  • Verify the ClusterResourceSet feature is actually ON after clusterctl init. CRS is gated by EXP_CLUSTER_RESOURCE_SET. If it's not enabled by default in our pinned CAPI (v1.13.x), the CRS is inert and the user is stuck again with no error. If needed, document export EXP_CLUSTER_RESOURCE_SET=true before clusterctl init (or bake it into clusterctl.yaml). Confirm the CRS controller pod is running as part of install-provider.md §4 verify.
  • POD_CIDR coupling. The bundled Calico pool is hardcoded 10.244.0.0/16. If a user overrides POD_CIDR, the pool won't match and pods misroute. Document the coupling in templates.md; ideally add a preflight/validation note.
  • Namespace. CRS + ConfigMap must live in the namespace where the workload Cluster objects are (default). Make the apply step namespace-correct and call it out.
  • Egress / air-gapped. CRS applies the manifest; Calico images (docker.io/calico/*:v3.28.2) must be pullable. Fine for stock clusters; note the mirror requirement in the air-gapped guide.

Acceptance

A user on a fresh machine, installing only from a release via the getting-started docs (no repo clone), gets a workload cluster that reaches Ready and schedules pods with zero manual CNI steps beyond the one documented one-time CRS apply — and the docs set the expectation so a transient NotReady never reads as a failure.

Related: #13 (closed) (templates hardcode location-specific image ids) — same "first-run must be smooth" theme.