CSI: provisioned PVs have no nodeAffinity and no attach enforcement — RWO volumes are unsafe / silently stuck on multi-worker clusters
Risk
A ReadWriteOnce volume backed by a whitesky.cloud disk works today only because test clusters have a single worker. On a multi-worker cluster the scheduler has no signal to keep volume-dependent pods together, and there's no clean single-attach enforcement — so a second pod can be placed on a different node and either hangs silently or (worst case) risks concurrent attach / data corruption. This is a data-safety issue, not just UX.
Evidence (live, my-cluster on try.whitesky.cloud)
- Provisioned PV has no
nodeAffinityat all —kubectl get pv <pv> -o jsonpath='{.spec.nodeAffinity}'is empty. So the scheduler's VolumeBinding plugin has nothing to constrain placement, even though nodes are labeledcloud.whitesky/location=be-mac-dc01-002(the driver's topology key). - Access mode
ReadWriteOnce; the disk attaches to exactly one VM (/dev/vdb). - The driver reports no
PUBLISH_UNPUBLISH_VOLUME("not watching VolumeAttachments") → Kubernetes' attach/detach-controller Multi-Attach protection never engages (noVolumeAttachmentobjects tracked). Attach is done node-side atNodeStageVolume. - (Compounded by #21 (closed): there's no
CSIDriverobject for the real driver namecloud.whitesky.csi, so attach semantics fall back to defaults.)
Failure scenario
- Deployment with
replicas: 2(or any two pods) sharing one RWO PVC, cluster with ≥2 workers in the location. - Scheduler places replica A on node-1, replica B on node-2 (nothing stops it — no PV nodeAffinity, no anti/co-affinity).
- The disk is attached to node-1's VM. Node-2's
NodeStageVolumecan't get it → replica B stuck inContainerCreatingwith a low-signal mount error, not a cleanMulti-Attach error for volume. - If the platform ever permits re-attach-while-in-use, that's concurrent RWO access → filesystem corruption.
Same trap on a reschedule: a pod moving to a different node relies on the old VM cleanly detaching first.
Fix (driver-side, like #21 (closed) — the CSI driver is a reused external component)
-
Set PV topology /
nodeAffinity. The driver should returnaccessible_topologyfromCreateVolume/NodeGetInfoso provisioned PVs carry anodeAffinityoncloud.whitesky/location— at minimum confining pods to the correct location, and giving the scheduler a real volume-topology signal. -
Enforce single-node attach. Either implement
ControllerPublishVolume/ControllerUnpublishVolumeso Kubernetes tracksVolumeAttachments and raises a clean Multi-Attach error (and blocks the unsafe second mount), or guarantee node-side that a disk attached to another VM is refused with a clear, surfaced error — never silently, never with a chance of double-attach. -
Ship the correct
CSIDriverobject (ties into #21 (closed)) so attach behavior is explicit, not defaulted. -
Docs: until fixed, warn that a multi-replica Deployment sharing one RWO PVC is unsafe across nodes; steer stateful multi-replica workloads to a StatefulSet (
volumeClaimTemplates, one disk per pod), and single-owner volumes toreplicas: 1/strategy: Recreate.
Acceptance
On a multi-worker cluster: a RWO volume is never mounted from two nodes at once (enforced, not by luck); a second consumer gets a clear, surfaced error (or is co-scheduled by topology) rather than a silent hang; and there is no path to concurrent attach / corruption. Provisioned PVs carry a topology nodeAffinity.
Related: #21 (closed) (driver/CSIDriver name mismatch), #20 (closed) (PV walkthrough), #17 (closed)/#18 (closed) (e2e — the multi-worker case should be exercised).