Kubernetes for AI mostly means one thing: running an LLM on GPU nodes without the cluster ignoring the GPU. You get there by making GPUs visible to the scheduler as the resource nvidia.com/gpu through the device plugin framework, requesting them in the pod with requests and limits, and running the inference server as an ordinary Deployment with probes and autoscaling. Kubernetes LLM workloads turn out to differ less from other workloads than you might expect, only the resource is new and the failure modes are different.
I run production Kubernetes clusters on Hetzner and I am a CNCF Kubestronaut, all five certifications. None of those clusters carry GPUs yet: my Qwen3.8 model runs on its own Nvidia DGX Spark, outside Kubernetes. That contrast, a production cluster here, a standalone server there, is a good way to see when each setup earns its complexity. You can find all my Kubernetes articles on the Kubernetes page.
GPU nodes in the cluster: device plugin and node feature discovery
To Kubernetes, a GPU is invisible by default. CPU and memory are native resources the scheduler already understands, a graphics card is not. It becomes visible only through the device plugin framework: a vendor-specific device plugin runs on each node, registers with the kubelet, and advertises the GPUs it finds as a custom resource, nvidia.com/gpu for Nvidia hardware. Without that plugin the scheduler simply does not see the GPU, no matter how many are physically installed.
In a cluster with several GPU types, node feature discovery adds another layer. It detects hardware characteristics automatically and turns them into node labels; the GPU-specific ones, such as memory size and card model, come from GPU feature discovery on top of it, which the GPU Operator ships along. You can then target those labels with node affinity in the pod spec, for example to make sure a large model only lands on a node with enough memory. The interplay of device plugin, labels and scheduler is documented in the Kubernetes documentation on GPU scheduling.
The GPU Operator: Nvidia software without manual work
Installing drivers, container runtime, device plugin and monitoring on every GPU node by hand is tedious and error prone, especially once nodes come and go through autoscaling. That is exactly what the Nvidia GPU Operator is for: once you deploy it, it installs and maintains the driver, device plugin, container toolkit, automatic node labeling and DCGM-based monitoring. New GPU nodes end up being treated much like CPU nodes, without you building a custom GPU image.
For a single, stable GPU server that effort often is not worth it, a manual driver install works fine there. Once several GPU nodes share the same cluster and keep changing, though, the Operator saves real manual work. Details and installation paths are in the GPU Operator repository.
Requesting GPUs: nvidia.com/gpu in requests and limits
Once the scheduler can see GPUs, you request them in the pod like any other resource, with one twist: GPUs belong in limits, not necessarily in requests as well. If you only set a limit, Kubernetes copies that value into the request automatically. If you set both, the two numbers must match, a request without a matching limit is not allowed. Unlike CPU, GPU requests are always whole numbers, there is no such thing as half a GPU without an extra mechanism.
apiVersion: v1
kind: Pod
metadata:
name: qwen-inference
labels:
app: qwen-inference
spec:
containers:
- name: sglang-server
image: my-registry/sglang-server:latest
resources:
limits:
nvidia.com/gpu: "1"
memory: "32Gi"
cpu: "4"
ports:
- containerPort: 30000
If the scheduler cannot find a node with enough free nvidia.com/gpu capacity, the pod stays Pending, exactly like it would for missing CPU or memory. How Kubernetes evaluates requests and limits in general, and where the common traps sit, is covered in Kubernetes Requests and Limits Explained, and the same logic applies to GPUs.
Sharing one GPU across pods: time-slicing and its limits
GPUs are expensive, and not every workload needs a whole card to itself. The GPU Operator offers time-slicing for that: through a ConfigMap with a replicas field, a node advertises more GPU resources than are physically present, four instead of one with replicas: 4, for example. Several pods then share the same physical GPU on a rotating time basis.
The catch matters: time-slicing gives you no memory or fault isolation between the pods, and a pod that requested more replicas does not get a larger compute share, all replicas split the time evenly. For short, bursty jobs or development environments that is a reasonable trade-off. For a single large language model whose weights alone already occupy most of the GPU memory, my Qwen3.8 model with its NVFP4 checkpoint of roughly 22 gigabytes for instance, time-slicing makes little sense: memory is the bottleneck there, not compute time.
The LLM server as a Kubernetes Deployment: probes and model storage
A single pod does not get replaced after a crash. An LLM server therefore belongs in a Deployment, which keeps the desired number of replicas running and replaces a crashed pod automatically. Two things set LLM servers apart from a typical web app here.
First, the probes: a language model can take several minutes to load its weights and start answering, far longer than a web server needs to boot. Set initialDelaySeconds or failureThreshold too tight, and Kubernetes kills the container mid-load, before it ever got a chance. vLLM, for example, ships its own /health endpoint that you point readiness and liveness probes at, with generous startup allowances.
Second, storage: model weights run into several gigabytes, and nobody wants to re-download them on every pod restart. A persistent volume claim for the model cache solves that: the pod mounts the PVC instead of downloading again at every start. If the server runs across several nodes at once, the volume needs an access mode that allows concurrent reads from multiple nodes, or you keep a local copy per node.
Scaling by load, not by CPU
Standard HPA measures CPU utilization, and that tells you almost nothing for LLM inference: the GPU is the bottleneck, while an inference pod's CPU usage often sits at a few percent regardless of how full the queue is. More useful metrics are requests per second, internal queue length, or GPU utilization pulled through an exporter like DCGM and wired up via a Prometheus adapter or KEDA. How HPA works in general, which metrics it supports and where the usual pitfalls sit, is covered in Kubernetes Autoscaling with HPA.
One server or a cluster: when a DGX Spark is enough
Not every LLM workload needs Kubernetes. My Qwen3.8-27B runs under SGLang directly as a systemd service on a Nvidia DGX Spark, no cluster, no device plugin, no HPA. What I measured: roughly 50.7 tokens per second on decode. That number holds for this exact setup, the NVFP4 checkpoint of Qwen3.8-27B under SGLang on this one machine, with a single request and no batching. It does not transfer to a different model, a different quantization or several concurrent requests. For that scenario, a Kubernetes cluster would be pure overhead nobody actually uses.
A cluster with GPU nodes earns its complexity once several models or versions need to run in parallel, load fluctuates and should scale automatically, several teams or customers share the same server pool, or a node is allowed to fail without taking the service down. That is the same trade-off I already weigh in my GPU-free production clusters, just with a pricier resource on the scale.
| Criterion |
Standalone server (e.g. DGX Spark) |
Kubernetes cluster with GPU nodes |
| Models in parallel |
one, permanently loaded |
several, per deployment or namespace |
| Scaling under load spikes |
none, fixed capacity |
HPA adds replicas across nodes |
| Fault tolerance |
one server, one point of failure |
a node failure gets rescheduled automatically |
| Operational effort |
one systemd service |
GPU Operator, monitoring, storage, RBAC |
| Typical case |
one person or small team, one fixed model |
several teams, shifting workloads |
Where the line sits for you depends on how many users you serve and how much downtime you can accept. How SGLang, Qwen3.8 and the DGX Spark actually perform for me, measurements included, is in Qwen 3.8 Locally: DGX Spark Experience.
Frequently asked questions
Do I always need the Nvidia GPU Operator for GPU nodes in Kubernetes?
No. For a single, stable GPU node, a manually installed driver with the device plugin is often enough. The GPU Operator pays off once nodes appear or get replaced automatically, in autoscaling clusters for instance, because it sets up driver, plugin and monitoring on every new node by itself.
Can a pod get half a GPU?
Not by default: nvidia.com/gpu is always requested as a whole number. Shared use only works through extra mechanisms like time-slicing or Multi-Instance GPU, both with their own trade-offs around isolation and performance.
Does HPA scale an inference server sensibly by CPU usage?
Barely. In LLM inference the GPU is the bottleneck, and CPU load often stays low no matter how busy the server actually is. Custom metrics such as requests per second or GPU utilization through an exporter are far more informative.
Does the model have to reload on every pod restart?
Only if the weights are not sitting on a persistent volume. With a PVC for the model cache, a freshly started pod just mounts the file instead of downloading it again, which for several gigabytes is the difference between seconds and minutes.
Where to go from here
Requests, limits, probes and autoscaling are not GPU-specific by nature, they apply to every Deployment, GPU or not. All of that, with full examples, is covered in chapter 8 of my Kubernetes Practical Guide (Rheinwerk Computing). Find everything about the book on my Kubernetes page.
My suggestion for the next step: if you already run a Kubernetes cluster, check first whether you actually need a GPU node before rolling out the GPU Operator, a single server might be enough to start. Details on the serving side are in the vLLM documentation on Kubernetes deployment.
