GPU: pre-baked node images per vendor (NVIDIA / AMD / Intel) and a vendor-neutral GPU path

Summary

Ship pre-baked node images carrying GPU drivers, for more than one vendor — NVIDIA today, AMD and Intel next — and generalise the GPU path so a second vendor is configuration rather than a fork.

Why this is bigger than images

#34 (closed) delivered GPUs end to end, but every layer above the attach is NVIDIA-shaped:

Layer Today AMD Intel
Driver install (spec.gpu.driverVersion) nvidia-driver-<branch>-server from Ubuntu's archive ROCm (amdgpu-dkms), separate repo intel-i915-dkms / in-tree, plus compute runtime
Container runtime hook nvidia-container-toolkit + containerd nvidia runtime none needed (device nodes + groups) none needed
Device plugin nvidia-device-plugin, advertises nvidia.com/gpu amdgpu-device-plugin, amd.com/gpu intel-device-plugins, gpu.intel.com/i915
Pod request resources.limits: nvidia.com/gpu amd.com/gpu gpu.intel.com/i915
RuntimeClass nvidia not used not used

So "add AMD" is not "add an image" — it is a second driver recipe, a second device plugin, a different resource name, and no RuntimeClass. Shipping an AMD image without the rest would produce a node with a GPU that nothing can allocate.

The good news: the backend already tells us the vendor

GPUProfile carries vendor alongside model (pkg/vco/gpu.go), and the controller already resolves the profile before attaching. So the provider can know it just attached an AMD card and act accordingly, rather than making the user restate it:

type GPUProfile struct {
    ID, Model, Vendor, Description string
    AvailableInstances int
}

That makes a vendor-neutral design cheap: derive the vendor from the resolved profile, and let it select the driver recipe, the add-on and the node label. spec.gpu.driverVersion stays a per-vendor version string; only its interpretation changes.

Scope

1. Vendor-neutral provider path

  • Derive vendor from the resolved profile; record it in status.gpus[].vendor.
  • Node label carries vendor as well as model, so a pod in a mixed cluster can target either.
  • cloudinit.MergeNvidiaDriver becomes vendor-dispatched (MergeGPUDriver), with the NVIDIA recipe as the first implementation. Each recipe keeps the current contract: install before kubeadm, and end in a probe (nvidia-smi -L / rocm-smi / clinfo) so a broken install fails the node's boot rather than yielding a Ready node whose GPU nothing can use.
  • Reject the combination we cannot serve — an AMD profile with an NVIDIA-only add-on enabled — loudly, at reconcile, listing what is supported.

2. Device-plugin add-ons per vendor

config/addons/gpu/ becomes vendor-selectable via the enablement label (…/gpu: nvidia | amd | intel), each bundle shipping that vendor's plugin and, where relevant, its RuntimeClass. The label is already the selector, so this is additive.

3. Pre-baked images (the original ask)

Extend image-builder/ with GPU variants per (vendor, OS, k8s version), carrying driver + userspace + container hooks:

  • NVIDIA — the datacenter driver, nvidia-container-toolkit, containerd nvidia runtime. Also the ONLY route for a mediated vGPU, whose licensed guest driver is in no public archive (see docs/known-limitations.md).
  • AMD — ROCm/amdgpu, matched to the card generation.
  • Intel — i915 plus the compute runtime.

Also bake a nouveau blacklist into the NVIDIA variant: on a stock image nouveau binds the card at boot, and while the live install now unloads it, a baked blacklist removes the failure mode entirely (#34 (closed)).

Cost warning. This multiplies the matrix that #25 and #28 already multiply: vendor × OS × k8s version. Build-once/skip-if-published (#16 (closed)) is a hard prerequisite, and a GPU image is only worth building for combinations someone actually asks for — an explicit list, not a cross product.

4. Validation

Extend the gpu feature-matrix scenario's workload tier to be vendor-parameterised (resource name and probe command per vendor), so each vendor is proven the same way rather than assumed to work by analogy.

Open questions

  • Which vendors are actually in the fleet, and in which locations? Today's hardware is a stub profile in one location, so none of the vendor paths are exercised. Build for what exists.
  • AMD/Intel passthrough behaviour in the guest — driver binding, reset behaviour on VM restart, whether large-BAR/UEFI applies as it does for NVIDIA datacenter cards.
  • Licensing: NVIDIA vGPU guest drivers require entitlement; ROCm and Intel do not. That asymmetry decides whether an image can be built in CI at all or needs a credentialed build (compare the RHEL/SLES subscription problem in #25).

Acceptance criteria

  • Vendor is derived from the resolved GPU profile and recorded in status.gpus[]; the node label carries vendor and model.
  • The driver install is vendor-dispatched, each recipe ending in a probe that fails the boot on a broken install.
  • A device-plugin add-on exists per supported vendor, selected by the …/gpu label value.
  • Pre-baked GPU images published for the agreed (vendor, OS, version) list, built once and reused (#16 (closed)).
  • An unsupported vendor/add-on combination fails loudly at reconcile, naming what is supported.
  • The gpu e2e workload tier is vendor-parameterised and passes for each shipped vendor.

Follows #34 (closed) (GPU by type — the provider side, NVIDIA). Related: #25 (OS matrix — same multiplication), #16 (closed) (build-once, a prerequisite), #28 (version currency), #35 (feature-matrix completeness).