> ## Status after v0.5.0 — narrowed to the workload tier > > **The provider side shipped in !12.** A pool asks for a GPU by type, the provider resolves it per location, reserves and attaches, records what it holds, and releases everything on teardown — including reservations left by a machine deleted with `--force`, reclaimed by an owner-encoded reservation name. Proven end to end against a real backend by the `gpu` e2e scenario. > > **What remains needs GPU hardware this platform does not offer.** Only stub `Dummy` profiles exist here: they attach, but there is no device behind them, so a driver, a device plugin and `nvidia-smi` cannot be exercised at all. That is the entire remaining scope of this issue. > > Delivered: > - [x] request GPUs by **type + count**; no instance id in user-facing config > - [x] resolve per location, reserve deterministically named instances, attach before first boot > - [x] `status.gpus` reports the instances; an absent/exhausted type fails loudly with the location's actual types > - [x] machine **and** cluster deletion release every reservation — verified zero `unassigned_vgpus`, plus the force-delete path > - [x] `gpu` scenario in the feature-matrix e2e, including the release-on-teardown assertion > - [x] `docs/guides/gpu.md` > > Remaining (blocked on hardware): > - [ ] a GPU node image with driver + container toolkit + runtime class — **superseded**: see #37, which argues the vendor GPU Operator should be the primary driver path instead, and #36 for the image matrix > - [ ] nodes advertise `nvidia.com/gpu` via the device-plugin add-on (the bundle ships and is label-gated; unverified without a device) > - [ ] a pod requesting `nvidia.com/gpu: 1` runs and sees the device > > Partially delivered: > - Geo per-site resolution is implemented (the machine's own location decides the profile) but **not** covered by an e2e — the geo scenario provisions no GPUs. > > Deviations from the proposal below, both deliberate: > - The GPU is attached **after** the VM is created but **before** it is started, not via `VMCreateExtensions`. Same guarantee — the device is present on first boot, so the driver and device plugin see it rather than a device that appeared later — and it keeps one code path for both the create and the repair case. > - The device-plugin add-on is **not** wired into `addonLabelChangedPredicate`. That predicate exists to re-run credential injection when an add-on is enabled on a live cluster; the GPU add-on needs no credential, so it has nothing to re-run. > > Several open questions below were answered by real runs — recorded in !12 and `docs/guides/gpu.md`: > - `GET …/locations/{loc}/gpus` returns **one profile per physical card**, with the PCI address in the model (`Dummy GPU 0000:f1:e1.0`). A type therefore spans several profiles, so the resolver matches in tiers and picks the emptiest card with room. > - A vGPU **cannot** be detached from a running VM, and `detach_gpu=true` on the VM delete does not exempt it — teardown stops the VM, deletes it, then releases. > - `count > 1` works by repeating reserve+attach; capacity is checked per card before reserving. > - A reservation is billed while merely **reserved**: detaching does not stop the charge, it only makes it unassigned. ## Original issue ## Summary whitesky.cloud supports GPUs / vGPUs in VMs, but CAPVCO does not expose them: there is no GPU field in `VCOMachine`/`VCOMachineTemplate`, the VM-create call never sends `gpu_id`/`vgpu_name`, the node images carry no driver or container runtime hook, and no device plugin is shipped. We want a workload cluster's pods to consume GPUs **by type**, provisioned declaratively and released cleanly. ## Background — what the platform provides, and the abstraction gap The platform's GPU API is **instance-oriented**: you reserve a *named* vGPU instance out of a GPU profile and then attach that specific `vgpu_id` to a VM. There is no "give me one A100" primitive. Relevant calls (`swagger.json`): | Call | Meaning | |---|---| | `GET /customers/{cid}/locations/{loc}/gpus` → `GPUProfileStructs` | `{id, model, vendor, description, available_instances}` | | `POST /customers/{cid}/locations/{loc}/gpus/{gpu_id}` body `{vgpu_name}` → `{vgpu_id}` | reserve a vGPU from a profile | | `POST /…/cloudspaces/{cs}/vms/{vm}/gpus/{gpu_id}` body `{vgpu_name}` → `{vgpu_id}` | reserve **and** attach in one call | | `POST` / `DELETE /…/vms/{vm}/vgpus/{vgpu_id}` | attach / detach an existing reservation | | `GET /customers/{cid}/locations/{loc}/vgpus` | list reservations: `{vgpu_id, vgpu_name, gpu_id, status, virtual_machine{…}}` | | `DELETE /customers/{cid}/locations/{loc}/vgpus/{vgpu_id}?detach=&permanently=` | release a reservation | | `VMCreateExtentions{gpu_id, vgpu_name}` | GPU attached at VM birth | **A user is never interested in a specific GPU instance** — only in pods getting a GPU of a given type. Turning "type + count" into "reserve these named instances, attach them, track them, release them" is the core of this feature, not an incidental detail. Note also `AccountConsumptionModel.unassigned_vgpus[]`: a reservation that is not attached **is still billed**. Lifecycle correctness is therefore a functional requirement — a leaked vGPU costs money, silently. ## Goal 1. **Declarative, type-based request** — a `VCOMachineTemplate` asks for *N GPUs of type X*; the provider resolves, reserves, attaches and releases the concrete instances. 2. **Pods get GPUs by type** — nodes advertise `nvidia.com/gpu`, and the GPU *type* is discoverable as a node label so a mixed-type cluster can target one. 3. **No leaks** — deleting a machine or cluster releases every reservation it created. ## Proposed shape (open for design) ### 1. Provider API — type in, instances hidden ```yaml spec: gpu: type: "NVIDIA A100" # vendor+model match (case-insensitive), OR profileId: "<gpu-profile>" # exact escape hatch count: 1 ``` Template variables (`VCO_GPU_TYPE`, `VCO_GPU_COUNT`) on a `gpu` flavor or the existing templates, plus CRD regeneration. GPU workers are their own `MachineDeployment`, so a cluster can mix a CPU pool and one pool per GPU type. ### 2. Resolver + reservation — the abstraction layer - **Resolve** `type` → profile id **per location**, by grouping the location's GPU listing on `(vendor, model)` and checking availability. Grouping client-side means the resolver is correct whether the listing is profile-level or instance-level (see open questions). No match → fail loud, listing the types the location actually offers, per the `VCO_IMAGE_ID` convention. - **Reserve** with a deterministic name — `capvco-<cluster>-<machine>-gpu-<n>` — so a reservation is traceable to its owner, idempotent on retry (look up by name before reserving, or every requeue leaks a billed vGPU), and reclaimable by an orphan sweep. - **Attach at create** via `VMCreateExtensions{gpu_id, vgpu_name}` so a GPU node is born with its GPU — no post-create mutation, consistent with immutable machines. Reserve-then-attach endpoints are the repair/reconcile path only. - **Record** `status.gpu[]` = `{vgpuId, profileId, model, attached}`, mirroring `status.etcdDiskId`/`etcdDiskAttached`. - **Geo**: profile ids are location-scoped like image ids — per-site resolution following the `GeoSpec.LocationImages` pattern. ### 3. Lifecycle / leak prevention - Machine delete (existing `MachineFinalizer`): detach, then `DELETE …/vgpus/{vgpu_id}?detach=true` — the API has an explicit "detach before deleting" flag. - Reconcile-time **orphan sweep**: list the location's vGPUs, release any carrying this cluster's name prefix whose machine no longer exists; warn loudly, as the e2e harness does for cloudspaces and disks. - Deleting a cluster must leave **zero** `unassigned_vgpus` behind. ### 4. Node image (`image-builder/`) A GPU variant: NVIDIA driver matching the vGPU profile, `nvidia-container-toolkit`, and the containerd `nvidia` runtime class. Pre-baked is the established path and the only one that works air-gapped. This multiplies the image matrix — coordinate with #25. ### 5. Device-plugin add-on (`config/addons/`) An `nvidia-device-plugin` ClusterResourceSet, label-gated (`capvco.infrastructure.cluster.x-k8s.io/gpu: nvidia`) and wired into `addonLabelChangedPredicate` exactly like `csi` / `ccm` / `service-lb`, so enabling it is a `kubectl label` like every other add-on. Node labels carry vendor/model for type-aware scheduling (device plugin, optionally node-feature-discovery). ### 6. Docs + validation - `docs/guides/gpu.md`: pick a type, size a pool, request `nvidia.com/gpu` in a pod, target a type. - A `gpu` scenario in `hack/e2e-feature-matrix.sh`: provision a GPU worker → node advertises `nvidia.com/gpu` → an `nvidia-smi` pod sees the device → tear down → **assert the reservation is released** (the billing-leak guard). ## Open questions - **What does `GET …/locations/{loc}/gpus` actually return?** `swagger.json` describes GPU *profiles* with `available_instances` (type + count), but the observed behaviour is a listing of individual available GPUs. This decides whether the resolver merely filters or must group and count — the proposal groups client-side so it works either way, but the real shape should be confirmed against a live location. - Passthrough (`/vms/{vm}/gpus/{gpu_id}`) vs vGPU semantics — is a whole-GPU passthrough expressible, and does it differ from a vGPU reservation? Both take the same `{vgpu_name}` body. - Can a vGPU attach to a **running** VM, or must it be stopped? If create-time only, the create-time path is mandatory rather than merely preferred. - What is `POST /customers/{cid}/locations/{loc}/vgpus/{vgpu_id}` (no body, 200) — restore a deleted reservation? - Is `count > 1` per VM supported (repeat reserve+attach), and are there per-VM or per-profile caps? - vGPU **profile sizing** (frame-buffer split): does a profile already encode the slice, or is that a separate parameter the API surfaces elsewhere? - Quota/availability races: two machines reserving the last instance concurrently — is there an error to retry on, or must the provider pre-check? ## Acceptance criteria - [ ] `VCOMachine`/`VCOMachineTemplate` request GPUs **by type + count**; no instance id appears in user-facing config. - [ ] The provider resolves the type per location, reserves deterministically named instances, and attaches them at VM create. - [ ] `status.gpu[]` reports the attached instances; a missing/unavailable type fails loudly with the available types listed. - [ ] Geo clusters resolve the type per site (`LocationImages` pattern). - [ ] Machine/cluster deletion releases every reservation — verified zero `unassigned_vgpus`. - [ ] A GPU node image (driver + container toolkit + runtime class) is built and published. - [ ] A label-gated `nvidia-device-plugin` add-on makes nodes advertise `nvidia.com/gpu`; GPU type is exposed as a node label. - [ ] A pod requesting `nvidia.com/gpu: 1` runs and sees the device (`nvidia-smi`). - [ ] `gpu` scenario in the #18 feature-matrix e2e, including the release-on-teardown assertion. - [ ] `docs/guides/gpu.md` documents the end-to-end path. Feature request, continuing the track of #24 / #25 / #27. Related: #27 (SR-IOV — same two-layer infra + node-plugin shape), #25 (node-image OS/version matrix, which a GPU variant multiplies), #18 (feature-matrix e2e), #28 (version currency).