My 96 GB RTX PRO 6000 Blackwell Max-Q now serves local inference alongside a 24 GB RTX PRO 4000 Blackwell SFF. Both cards share one Proxmox host, each passed through to its own K3s worker VM. Getting there exposed problems in PCIe topology, firmware assumptions, VFIO configuration, GPU scheduling, model loading and driver upgrades.
The 96 GB service answers chat and agent requests through LiteLLM. Firmware behavior, native IOMMU grouping, configuration drift and upgrade handling still need follow-up. This is a configuration and troubleshooting case study with a checklist, assuming familiarity with Proxmox and Kubernetes.
Jump to the onboarding checklist, firmware and passthrough, or K3s runtime and scheduling.
The consolidated environment, as inspected after the move:
| Component | Recorded configuration |
|---|---|
| Motherboard and firmware | ASUS Pro WS W680-ACE, AMI BIOS 4601 |
| CPU | Intel Core i7-13700T, 16 cores / 24 threads |
| Host memory | 192 GB: four 48 GB DDR5 modules, running at 4200 MT/s, non-ECC |
| Hypervisor | Proxmox VE 9.2 |
| GPU guests | Two VMs, each with 60 GiB RAM, q35, OVMF and host CPU type |
| Guest OS and Kubernetes | Debian 13, K3s v1.36.2 |
| Guest NVIDIA driver | 615.71.09 |
| Main inference service | Qwen3.5-122B-A10B-NVFP4 on vLLM v0.28.0 |
Memory ballooning is disabled for both GPU guests. The cards are assigned as PCIe devices; the inspected VM configurations do not contain a custom QEMU MMIO-window override. The earlier model comparisons below belong to the AORUS build, not this software and hardware snapshot.
Why I chose the 300 W RTX PRO 6000#
The exact card is the NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition. It joined the RTX PRO 4000 Blackwell SFF Edition from my earlier GPU-node build. The edition names matter: both are Blackwell cards, but their power, memory and PCIe interfaces differ.
| Specification | RTX PRO 6000 Blackwell Max-Q | RTX PRO 4000 Blackwell SFF |
|---|---|---|
| GPU memory | 96 GB GDDR7 with ECC | 24 GB GDDR7 with ECC |
| Memory bandwidth | 1,792 GB/s | 432 GB/s |
| CUDA cores | 24,064 | 8,960 |
| Total board power | 300 W | 70 W |
| PCIe interface | Gen5 x16 | Gen5 x8 |
| Form factor | Full height, dual slot, 10.5 inches long | Half height, dual slot, 6.6 inches long |
| Cooling | Active | Active |
These are NVIDIA’s published specifications, from the Max-Q datasheet and SFF datasheet, not measurements of my workloads. Memory bandwidth is local to the GPU; it is separate from PCIe bandwidth. The exact Max-Q also has a TechPowerUp database entry as a supplementary reference.
The standard RTX PRO 6000 Blackwell Workstation Edition has a 600 W power budget. Both editions have 96 GB of GDDR7 ECC memory and 1,792 GB/s memory bandwidth. NVIDIA lists 110 versus 126 TFLOPS FP32, and 3,511 versus 4,000 effective FP4 AI TOPS with sparsity, for Max-Q versus Workstation. Those are peak specifications at GPU Boost clocks. The larger Workstation card uses double-flow-through cooling; NVIDIA describes Max-Q’s cooling as active.
There is also a separate Server Edition. My Max-Q is a workstation card. The listed performance gap did not justify a 600 W heat budget in my home setup: I wanted the 96 GB capacity, and the 300 W tradeoff made sense to me. I have not benchmarked both editions, so that choice is not a claim that they perform equally.
The plan before delivery#
Before delivery, I used OpenAI Codex and Anthropic Claude Code to research how to divide workloads between the 96 GB and 24 GB cards.
The first idea was to keep several model services resident on the large card and advertise multiple scheduling slots through time slicing. The revised baseline gave one main language-model engine the whole card, compared checkpoints one at a time, and kept the 24 GB card as a support tier. Extra scheduling slots do not create more VRAM or isolate one model’s allocations from another’s.
One old assumption survived into the arrival checklist: it still expected eight time-sliced slots, although the accepted plan required one. I corrected it during onboarding: the installed configuration still needed to be checked against the decision it was meant to implement.
One host, two cards, and an actual slot budget#
The GPUs started in separate physical hosts. The 96 GB card was on a Gigabyte Z790 AORUS board; the 24 GB card had its own host. I wanted to consolidate them, but the AORUS layout did not leave room for a second GPU at a useful link alongside the 10 GbE adapter connecting my hosts. Slot spacing and electrical wiring both counted. Another full-length slot is not necessarily another useful GPU slot.
I replaced the Gigabyte AORUS board with an ASUS Pro WS W680-ACE, keeping the CPU and memory. I also moved the build into a Chieftec rack case with 120 mm fans to give cooling proper attention.
Power needs its own checklist#
The Max-Q needs a PCIe CEM5 16-pin auxiliary power connection. NVIDIA’s installation guide specifies the included NVIDIA adapter, fed by two separate PCIe 8-pin cables from the PSU. Follow that connection procedure, use cables compatible with the exact PSU, and check the complete system power budget. The GPU-end connector must be fully seated.
My first PSU and adapter combination did not get the card working. Replacing that setup with a Seasonic PSU did. I did not isolate whether the original fault was the PSU, adapter or connection, so I cannot turn that experience into a rule that adapters do not work. The 70 W RTX PRO 4000 SFF had required no such cable: NVIDIA’s installation guide confirms that it takes no auxiliary power connection.
The ASUS board has another connector worth noticing: its six-pin PCIE_6P_PWR supplies additional
power to the PCIe slots. It serves a different purpose from the GPU’s own 16-pin input; the
motherboard manual documents it in the power-connectors section.
The consolidated build#
Both cards now share one Proxmox host. Each is passed through to its own K3s worker VM, and the former host for the smaller card is powered off. Consolidation makes the physical setup simpler, but it also puts both GPU services in one failure domain: two VMs do not provide protection from maintenance or failure of their shared host.
After the move, Proxmox reported roughly 93% host memory utilization, enough to fire the high-memory alert. That is a host-wide reading, not the sum of the two GPU guests’ configured 120 GiB. I did not capture a breakdown of other guests, host processes and caches, so the remaining consumption still needs attribution before capacity tuning.
What matters is how the slots are wired. The ASUS manual specifies two CPU-connected PCIe 5.0 slots: one card in the first slot gets x16; two cards run x8/x8. The other full-length slots are chipset-connected PCIe 3.0 x4. Choosing either of those would be a very different configuration.
The host inspection confirmed two CPU-connected ports, each capable of Gen5 x8. Both links were at idle Gen1 x8 when read. GPU telemetry also records Gen5 x8 after the move.
Open the topology diagram at full size.
This is why I bought the board. It can keep both cards on CPU-connected Gen5 links in the intended slots. NVIDIA specifies the SFF card’s interface as Gen5 x8, so that layout supplies its full specified width; the Max-Q card uses eight of its sixteen supported lanes. Intel’s bandwidth table puts Gen5 x8 at roughly the same theoretical one-direction bandwidth as Gen4 x16: about 31.5 GB/s before PCIe transaction overhead.
This also puts the width question from my earlier SFF post in context. The card advertises an x16 capability in its PCI registers, which monitoring reports as its maximum. Its specified interface is x8, and it has trained to x8 in both hosts: Gen4 on the 24 GB card’s former host, Gen5 on the ASUS. The new guest reports Gen5 with the same q35/OVMF arrangement. That corrects an ambiguity in the earlier post: the 24 GB card’s former physical platform imposed the Gen4 limit; virtualization itself did not. This was a separate machine from the 96 GB card’s AORUS host.
Check the link in Linux#
I identify the board on the physical host, then inspect the PCIe tree and the NVIDIA functions:
sudo dmidecode -s baseboard-manufacturer
sudo dmidecode -s baseboard-product-name
sudo dmidecode -s bios-version
lspci -Dnnk -d 10de:
lspci -tdmidecode reads firmware-reported inventory; it does not measure a negotiated link. In the tree,
find the GPU and its upstream bridge. Replace these example addresses with the ones from your host:
GPU_BDF=0000:01:00.0
UPSTREAM_BDF=0000:00:01.0
sudo lspci -s "$GPU_BDF" -vv | grep -E 'LnkCap:|LnkSta:|Region [0-9]:'
sudo lspci -s "$UPSTREAM_BDF" -vv | grep -E 'LnkCap:|LnkSta:'In lspci, LnkCap describes capability and LnkSta describes the negotiated state.
This is an excerpt from the ASUS upstream-port inspection, with addresses and unrelated fields omitted:
LnkCap: ... Speed 32GT/s, Width x8 ...
LnkSta: Speed 2.5GT/s ... Width x8 ...That is a Gen5-capable x8 port running at Gen1 x8 while idle. NVIDIA explicitly documents that link generation and width may be reduced when the GPU is not in use. In the guest, where the NVIDIA driver runs, I can sample the reported state while a normal model request is active:
nvidia-smi --query-gpu=name,pcie.link.gen.current,pcie.link.gen.max,pcie.link.width.current,pcie.link.width.max --format=csv --loop=1Stop the sampler with Ctrl-C, and repeat the host-side check during the same activity. The host shows the upstream path that a VM’s virtual PCI tree cannot establish. Compare current generation, width and both ends of the link before diagnosing a downgrade from one idle reading.
What PCIe speed changes for inference#
PCIe matters when moving data between host memory and GPU memory: loading weights, offloading parts of a model, and some multi-GPU communication paths. Once the weights sit in VRAM, the GPU reads them from its own memory for every token. NVIDIA’s CUDA guidance recommends keeping reused data on the device for precisely this reason.
Model choice, quantization, context, batching and memory use matter more than the link label for this resident-weight service.
The first startup after consolidation shows where the traffic is. The largest sampled PCIe receive rate was 1.76 GB/s while the model loaded. In the two busy serving hours examined afterward, the hourly maxima were about 12 MB/s into the GPU and 2 MB/s out. DCGM used a roughly 30-second collection cadence; Prometheus scraped the exporter every 15 seconds, so successive scrapes could repeat the same underlying sample. Short bursts can still be missed; the largest sample is not a guaranteed peak. It is about 6% of Gen5 x8’s theoretical 31.5 GB/s: I found no evidence of sustained link saturation in these samples. Offloading or splitting a model across cards would need its own measurements.
The telemetry adds the before-and-after picture. Checking width and generation together in the same samples, the 96 GB card reached Gen5 x16 before the move and Gen5 x8 afterward. The 24 GB card reached Gen4 x8 before and Gen5 x8 afterward. Each available hourly window contained at least one of those faster states; neither link was stuck at its idle generation.
The loading logs offer a useful comparison. Across twelve earlier starts, weight loading took 37.18-49.65 seconds. Four starts immediately following guest boots took 42.75-49.65 seconds; the first start after the move took 43.59 seconds, with the same model and vLLM image digest. The engine reported the same 73.24 GiB GPU memory footprint after loading. Dividing that footprint by 43.59 seconds gives about 1.80 GB/s, but allocated GPU memory is not a count of bytes transferred over PCIe, so that quotient cannot be compared with the sampled receive rate.
The practical result is that the one post-move load fell within the earlier post-boot range. The guest kernel also changed from 6.12.107 to 6.12.111; storage, cache state and deserialization were not isolated. Locating a startup bottleneck needs a profile of those stages and PCIe traffic.
Each bar is one restart from the Loki startup logs, in chronological order. An asterisk marks a start following a guest boot; cache state for the other starts is unknown. Only the orange ASUS bar belongs to the consolidated host.
Weight-loading values and download
| Restart | Host | Weight loading, seconds | Followed a guest boot |
|---|---|---|---|
| 1 | AORUS | 39.36 | Not marked |
| 2 | AORUS | 38.78 | Not marked |
| 3 | AORUS | 38.20 | Not marked |
| 4 | AORUS | 39.44 | Not marked |
| 5 | AORUS | 37.18 | Not marked |
| 6 | AORUS | 38.56 | Not marked |
| 7 | AORUS | 38.40 | Not marked |
| 8 | AORUS | 38.34 | Not marked |
| 9 | AORUS | 44.99 | Yes |
| 10 | AORUS | 42.93 | Yes |
| 11 | AORUS | 42.75 | Yes |
| 12 | AORUS | 49.65 | Yes |
| 13 | ASUS | 43.59 | Yes |
Download the 13 weight-loading records as CSV. Boot markers do not establish a cold cache.
Above 4G Decoding, Resizable BAR and the ASUS boot failure#
The old arrival checklist grouped Above 4G Decoding and Resizable BAR into one item, with both to be enabled. That was too casual.
- Above 4G Decoding permits suitable PCI resources to be mapped above the 4 GiB address boundary.
- Resizable BAR changes the size of a CPU-visible memory aperture. It does not add PCIe lanes or VRAM.
- VT-d and the IOMMU provide the address-translation and isolation machinery used for device assignment.
ASUS exposes these as separate settings; its Re-Size BAR option depends on Above 4G Decoding. The W680-ACE series BIOS manual and the Linux VFIO documentation describe different mechanisms here. The manual predates my BIOS 4601, so it explains the settings rather than proving their effective state in this build.
On the old AORUS board, Above 4G Decoding worked with the same CPU and memory. On ASUS, I had left it at its default, and the physical host would not boot. The onboard Q-Code display helped locate the failure during startup; disabling Above 4G Decoding got the machine booting again. I did not retain a reliable record of the exact code, so I cannot turn it into a diagnosis.
My working configuration now has Above 4G Decoding disabled, yet the host inspection shows a 256 MiB BAR1 aperture on each card, with both mapped above 4 GiB. The boot journal already finds both apertures there during initial PCI enumeration. Linux later reallocates the 24 GB card’s BAR1, still above 4 GiB, while the 96 GB card keeps its original mapping.
That does not fit a simple reading of the ASUS setting’s documented purpose. The remaining question is what the switch changes in this firmware build. A saved settings capture, including the separate Re-Size BAR state, and a controlled comparison would help answer it.
My arrival checks had ticked the BIOS/BAR item because the guest could see all 97,887 MiB of GPU memory. The post-move inspection reports that same capacity with a 256 MiB BAR1. Visible VRAM was the wrong measurement for checking a BAR aperture.
The failure was at host startup, so a guest MMIO workaround would not explain it. The cause remains open; I have no basis for a general recommendation to disable Above 4G Decoding or Resizable BAR.
The revised checklist records the exact setting, firmware version, BAR sizes, and where failure occurs: host startup, VM startup, driver initialization, or model loading.
Proxmox GPU passthrough: VFIO binding and IOMMU groups#
The physical host owns PCI assignment. The VM owns the NVIDIA driver and container runtime. Kubernetes only sees the GPU after both layers work.
The first automated VM creation failed because the account could not assign raw PCI devices. A later timeout left a partially created VM behind. Before retrying, I had to inspect the actual guest and reconcile automation state: a timeout did not mean that nothing had been created.
My Ansible Proxmox role enables the passthrough modules, manages driver binding and rebuilds the initramfs when that configuration changes. The K3s role then checks for the expected PCI device inside the guest before trying to validate the NVIDIA stack.
The two-card move exposed a particularly unhelpful configuration trap: repeated ids= assignments
to vfio-pci do not form a combined list. The last assignment wins. Adding a second card on a
second options line could silently leave only one card selected for VFIO.
For these exact two products, the managed options line contains both GPUs and their audio functions:
options vfio-pci ids=10de:2bb4,10de:22e8,10de:2c33,10de:22e9 disable_vga=1Those IDs identify these two products; other cards need their own. The Ansible role now owns
the complete /etc/modprobe.d/vfio-pci-ansible.conf file and removes superseded options from the
older shared file. One authoritative configuration is easier to verify than several additive edits.
IOMMU groups are a separate check. A successfully running VM proves that a device was assigned; it does not, by itself, document the isolation of every function in the group. I check group membership, the driver bound to each function, and whether unrelated host devices share that group. Linux VFIO treats the group as the unit of ownership.
For the GPU address selected above, these host commands show the group members and driver binding:
ls -l /sys/bus/pci/devices/"$GPU_BDF"/iommu_group/devices/
lspci -Dnnk -d 10de:
cat /proc/cmdlineInspect every member of the group, including the audio function. Check the kernel command line for an ACS override before describing those groups as native hardware isolation.
The post-move inspection shows both GPUs bound to VFIO. Each GPU function and each audio function
appeared in a separate IOMMU group, rather than one group per GPU/audio pair.
It also found pcie_acs_override=downstream,multifunction still on the host command line.
That option is absent from the current Ansible inventory, but the GRUB task adds required options
without removing pre-existing ones.
This is an unresolved configuration gap. I have not booted this build without the override, so the observed groups do not establish the board’s native isolation. Removing it and checking the result belongs in a separate host-maintenance window.
For the VM side, Proxmox recommends q35, OVMF and PCIe presentation for GPU passthrough. The guest still needs a compatible driver. Inside the VM, NVML reported the same link width and generation as the host. Only the host inspection shows the upstream port; neither reading measures throughput.
K3s GPU scheduling: runtime and device plugin#
Each layer has its own evidence. Seeing the card in the guest is necessary, but there are several more steps before a client can use it.
What actually installs the NVIDIA stack#
Ansible installs the guest’s NVIDIA driver and Container Toolkit. Argo CD deploys the NVIDIA device plugin, GPU Feature Discovery, Node Feature Discovery and DCGM exporter. I use these components separately; this cluster does not use GPU Operator to manage the stack.
Blackwell requires NVIDIA’s open Linux kernel modules. Inside the guest, this command identifies the loaded module’s flavor and version:
cat /proc/driver/nvidia/versionAfter an upgrade, distinguish the loaded module from the module file installed on disk.
| Layer | What I need to see before moving on |
|---|---|
| Proxmox and VFIO | GPU functions assigned to the intended VM, with the host’s grouping checked. |
| Guest driver | The expected GPU visible through nvidia-smi, after any required reboot. |
| K3s containerd | NVIDIA runtime usable by a workload, not merely a toolkit package installed. |
| Device plugin | The expected allocatable GPU count on the right worker. |
| Scheduling | A GPU request, RuntimeClass, node selection and taint toleration that agree. |
| Application | The exact checkpoint loads, becomes ready and answers through the normal client route. |
The 96 GB node uses the whole-card profile: one allocatable nvidia.com/gpu, rather than eight
time-sliced scheduling slots. The device-plugin configuration still contains alternative profiles;
their presence in a ConfigMap does not mean that a node selected them.
The 24 GB worker currently selects a four-slot time-slicing profile for support workloads.
Those slots share the same physical memory and do not provide four isolated 24 GB cards.
Separate VMs make that placement predictable. The deployed NVIDIA device plugin, v0.20.0,
disables custom resource names: two different cards in one worker would enter the
same nvidia.com/gpu resource pool. A request for one GPU
would not select 96 GB rather than 24 GB, and a node selector chooses the worker, not a particular
card inside it. NVIDIA documents per-node configuration and resource sharing.
Keeping one card per worker lets me select the large card’s whole-GPU profile independently
of the smaller card’s sharing profile.
Why I kept the large card whole#
Multi-Instance GPU, or MIG, divides a supported GPU into instances with dedicated memory and compute
resources. That is different from time slicing, where workloads share the physical GPU without
equivalent memory isolation. I considered MIG and deferred it until the baseline was measured.
The Max-Q supports
24 GB and 48 GB MIG partitions, but the current model’s 73.24 GiB GPU footprint does not fit
either. Its full-card 96 GB profile would leave no partition for another service.
My worker reports nvidia.com/mig.capable=false; NVIDIA’s
vBIOS and compute display-mode prerequisites remain unchecked on this build.
A simplified Pod-template fragment makes the scheduling contract visible. Its outer spec:
belongs directly under Deployment.spec.template:
spec:
runtimeClassName: nvidia
nodeSelector:
workload.ai/gpu-tier: big
tolerations:
- key: workload.ai/dedicated
operator: Equal
value: gpu
effect: NoSchedule
containers:
- name: inference
# Image, checkpoint, mounts, ports and probes belong here too.
resources:
requests:
nvidia.com/gpu: 1
limits:
nvidia.com/gpu: 1This is not a complete Deployment. The model disk, CPU and RAM budget, image, startup probe and network policy remain part of getting an engine to start. Kubernetes permits a GPU limit without an explicit request; it uses that limit as the request. If both entries are present, as here, their values must match.
Model weights need a disk before they need VRAM#
Before a model can load, its weights have to reach the node. The locked files for the revised candidate set totaled 282.78 GiB, against roughly 212 GiB in the original shortlist. Downloading every format in a multi-quantization repository would have required still more space; I needed the specific locked files.
I keep the weights on an additional model disk attached to the GPU VM, under /models/vllm,
and mount that directory into serving Pods through hostPath. This keeps model reads on the GPU
worker. I chose it with loading speed in mind, but did not measure it against NFS.
A prefetch Job on the other GPU worker does not populate the same cache. Before downloading, I check the mounted filesystem
and its free space on the exact worker that will load the model.
The early download-rate estimate also needed checking. Three sequential 256 MiB range requests averaged about 10.62 MB/s (10.13 MiB/s). During the actual parallel download, network telemetry recorded about 100.98 MB/s, with disk writes independently confirming the higher rate.
At those rates, the roughly 216 GiB multi-file portion projects to 38 minutes in parallel.
For the 66.8 GiB GGUF pair, the 1.9-hour estimate uses the single-file rate because that prefetch
Job fetched the two locked files sequentially with hf_hub_download.
These are planning estimates, not elapsed download times.
The vLLM prefetch Job was OOMKilled twice while configured with a 4 GiB memory limit. I raised its limit to 16 GiB. Disk fit, network throughput and downloader memory were three separate budgets.
If you restrict model-download egress, test the artifact’s redirects as well as the repository URL.
My Hugging Face download reached us.aws.cdn.hf.co. With Cilium 1.19.6, *.hf.co would not match it:
the single wildcard does not cross dots. My vLLM prefetch allowlist uses
huggingface.co, hf.co, **.huggingface.co and **.hf.co on TCP 443, plus DNS.
That is my configured allowlist, not a complete list of requirements for every Hugging Face download.
Also test a destination that must be denied; an overlapping broad allow policy can defeat the restriction.
Four defects the checks missed#
Rendering, schema validation, a server-side dry run and the repository tests passed. There were still four defect classes across two staged model deployments:
| Candidate | Failure | Where it became clear |
|---|---|---|
| GLM | A configured tool parser did not exist in the serving engine. | Engine startup. |
| Gemma | The memory budget left too little KV cache. | Startup validation, before a long request could run. |
| Gemma | The tool parser did not match the model’s emitted syntax. | A real tool-call check. |
| Both | Neither GPU requests nor GPU limits were set. | Inspection of the rendered manifests. |
A Pod with no GPU allocation is valid Kubernetes. In these two staged deployments,
nvidia.com/gpu was absent from both requests and limits; this was not a valid limits-only allocation.
A parser name in an application argument is
valid YAML. The missing piece was checking the behavior I actually wanted, with the exact image,
checkpoint and configuration together.
The comparison also needed deliberate replica ownership. Manually scaling a parked candidate up fought Argo CD’s desired count of zero, and the controller scaled it back down. For the comparison, replica management had to be explicitly delegated; for production, the desired running state had to be committed.
I had to repair the measurements before trusting the comparison#
One power-violation query reported a physically impossible 454% of wall time. Checking it
again while writing this post exposed a unit error: the DCGM field is in nanoseconds,
but the query divided by a microsecond-sized window. With the recorded numerator and the correct
10-minute denominator of 600e9 nanoseconds, the arithmetic gives 0.454%. This is a correction
of the recorded arithmetic, not a fresh measurement. I do not have an execution of the
corrected PromQL to publish; the related alert formulas still need a separate review.
There was a separate identity problem. DCGM exporter attached workload labels when a Pod claimed
the card and removed them when it left. Loading and unloading candidates therefore split one
device’s history across series. The configuration at the time used DCGM Exporter
4.6.0-4.8.3-distroless with Helm chart 4.8.3. I disabled optional Pod metadata enrichment and
removed selected attribution labels at scrape time. These are separate controls:
enablePodLabels: false does not disable Kubernetes device-to-Pod mapping.
Exporter label configuration
These are the relevant values under my chart’s dcgm-exporter dependency:
dcgm-exporter:
kubernetes:
enablePodLabels: false
serviceMonitor:
metricRelabelings:
- action: labeldrop
regex: "exported_(pod|namespace|container)"
- action: labeldrop
regex: "app_kubernetes_io_(name|instance|component)"The rendered configuration retains DCGM_EXPORTER_KUBERNETES=true and honorLabels: false.
The rules remove the listed attribution labels while preserving GPU identity and scrape-target labels;
they do not aggregate measurements. Optional Pod labels and device-to-Pod mapping are distinct in the
version-matched exporter implementation.
This exporter defaults virtual-GPU mapping to false, and my rendered values do not enable it. In that mode the mapping uses one Pod entry per device; the virtual-GPU/DRA paths can produce multiple Pod representations. Simultaneous sharing on the 24 GB worker remains unvalidated. If several representations become identical after label removal, fix the mapping or scrape strategy; summing physical-device measurements would count the same GPU more than once.
The dashboard also needed to distinguish the two cards. A fleet total can hide a missing target, and a blank error counter is not evidence of an error-free card. I want a positive scrape and temperature signal first, followed by per-card utilization, framebuffer use, power, temperature, slowdown information and PCIe link state.
An 85 °C cutoff was my rule, not the card’s verdict#
The initial comparison used six configurations, including two quantizations of the same smaller Qwen model. Under the then-current stop rule, the dense Gemma and T-pro runs were stopped with sampled peaks of 86 °C and 87 °C. The recorded thermal-violation counter remained zero. The accepted 122B candidate had peaked at 84 °C in that comparison, and staying below my cutoff had helped its selection.
I initially gave that too much meaning. The follow-up audit found 37 one-minute buckets at or above 85 °C across 2.6 days, spread across 14 hours of the card’s history, including the comparison runs, with no recorded thermal violations. Those are sampled buckets, not 37 continuously observed minutes or a measurement of the accepted model alone. I had not established a defensible reason for using that temperature as a selection boundary.
I withdrew that experimental cutoff. This did not remove monitoring or make heat irrelevant. It left the replacement acceptance threshold unresolved. The small comparison also confounded model architecture with power draw, so it did not establish that dense models were inherently unsustainable. These measurements also predate the new enclosure.
There was a similar correction in the kernel investigation. One NVFP4 checkpoint selected a weight-only fallback; another selected a native path on the same card. That was a difference in checkpoint and serving-path support, not proof that the GPU lacked native low-precision capability. I now record the selected kernel with the exact checkpoint instead of generalizing from the first log.
Local LLM throughput: method and limitations#
The recorded comparison repeated one Russian explanation prompt through LiteLLM, at temperature zero, with a 1,024-token output limit. Its throughput was reported completion tokens divided by the entire streaming request duration, including time before the first token. It was not isolated decode throughput or a concurrency test.
| Test configuration | Engine / selected path | Requests | Mean request throughput, tokens/s |
|---|---|---|---|
| Qwen3.6-35B-A3B, NVFP4 | vLLM / Marlin | 5 | 254.19 |
| Qwen3.6-35B-A3B, Q8_0 | llama.cpp | 5 | 200.00 |
| T-pro-it-2.1, Q8_0 | llama.cpp | 5 | 37.48 |
| GLM-4.7-Flash, BF16 | vLLM | 5 | 114.36 |
| Gemma-4-31B-it, BF16 | vLLM | 3 | 21.46 |
| Qwen3.5-122B-A10B, NVFP4 | vLLM / FlashInfer CUTLASS | 5 | 84.13 |
These are small, sequential smoke measurements on the 96 GB card’s AORUS host, not a sustained-capacity test. The accompanying correctness checks covered only a few prompts.
Historical candidate inputs and reproducibility limits
The historical candidate configuration contains all six checkpoint repositories and revision locks, both engine image digests, and configured context, memory and cache settings. These are Git configuration values from before promotion, not a verified record of the effective settings for each measured request. The configured vLLM version was 0.26.0; the later service snapshot near the opening uses 0.28.0.
The original Russian prompt, harness and individual request results were unavailable for this export. The earlier evidence review recorded them, but this download cannot replace those artifacts. Warm-up procedure and actual cache state could not be established. The vLLM candidate definitions enabled prefix caching; repeated identical prompts can reuse prefill work, affecting a metric that includes time before the first token. This limits comparison across configurations without invalidating the recorded observations. See vLLM’s prefix-caching explanation.
Promotion includes the things around the model#
The comparison ended with nvidia/Qwen3.5-122B-A10B-NVFP4 promoted to serve seven
chat and agent aliases. Embedding, reranking, speech and rollback routes remained separate.
I chose it over the faster 35B candidate for its native FP4 serving path, limited Russian-output
checks and usable request throughput. The previous service was Qwen3.6-35B-A3B UD-Q4_K_M on the
24 GB card, whose gateway baseline
reported a chat p50 of 86.3 visible tokens/s. That is context for the new result of 84.13 tokens/s,
not a matched comparison: the earlier test used ten requests capped at 128 tokens; this one reports
the mean of five capped at 1,024.
The 22-prompt language check counted Cyrillic output and English leakage; the planned manual quality scoring was never completed. The choice remains provisional pending a proper quality comparison.
The selected service needed more than a successful benchmark. Stopping the previous service while Git still declared it active left monitoring expecting an endpoint that I had deliberately removed. An always-on engine’s preallocated framebuffer pool also sat just above a global occupancy alert threshold. Reducing its configured memory fraction from 0.90 to 0.88 restarted the engine and interrupted requests for about two minutes. Alert tuning had become a serving change.
Autoscaling added another boundary. When I unpaused a candidate after its benchmark, KEDA still
saw traffic in the trigger’s increase(vllm:request_success_total[15m]) and kept it resident,
blocking the next GPU workload. Separately, the promoted service inherited the canary’s
minReplicaCount: 0 and 900-second cooldown, which could shut down the main service after idle
time. Both the trigger’s history and the scale-to-zero policy needed review before promotion.
I now review routing, probes, replica ownership, autoscaling and alerts together when promoting a model. A green Argo application is one observation. The useful finish line is a ready workload, a ready Service endpoint, and a request through the alias the client actually uses.
After consolidation, I traced the normal code alias
through LiteLLM and a ready Service endpoint to the Pod requesting the 96 GB GPU.
NVIDIA driver upgrades and NVML driver/library mismatch#
Driver and toolkit packages are managed through Ansible; Kubernetes components and serving images live in GitOps. A routine package upgrade exposed a gap between those layers. It installed NVIDIA userspace 615.71.09 while kernel module 610.57.04 stayed loaded. New GPU containers failed with an NVML driver/library mismatch. The existing 96 GB service continued answering requests and DCGM stayed green. That survival did not validate the upgraded stack; starting a fresh container exposed the mismatch.
The general update role watched /var/run/reboot-required, which this Debian DKMS rebuild did
not create. The driver-install role has its own reboot handling, but routine package upgrades
took the other path. I still need a guard that compares the loaded NVIDIA module with the installed
driver packages and exercises a fresh GPU container after an upgrade.
The post-move inspection found both guests on 615.71.09 with working GPU containers; the historical
mismatch is gone, while the upgrade path still needs that guard.
Serving-image updates have their own ordering constraints. For the move from vLLM
0.26.0 to 0.28.0, I updated the gateway first to handle the engine-output rename from
reasoning_content to reasoning, then deployed the release image pinned by digest. The previous
image identity stayed recorded as the rollback source.
The consolidation also left configuration work open. The VM definitions in Terraform still
describe the former two-host placement. Both live GPU VMs have onboot: 0, so starting the host
does not start those guests automatically. Bringing them up remains an explicit step.
For an update, I keep the previous package or image identity, change one layer at a time, and
repeat the short chain: device visible, runtime usable, GPU allocatable, engine ready, client
request succeeds, telemetry present. A reported CUDA compatibility version from nvidia-smi
does not establish which CUDA runtime an engine image actually contains.
What the new enclosure has shown so far#
This chart uses the DCGM power and temperature series behind my Grafana dashboards. It spans the move from the AORUS host to the consolidated build. I kept the gap instead of joining the line across missing observations.
Hourly means, UTC; each timestamp marks the end of the averaged hour. For each series, 323 of 360 hourly points are available; missing points remain gaps. The long gap coincides with the move, but includes a monitoring outage, so it cannot time GPU downtime precisely. Hourly averages also hide short temperature and power peaks.
Download the plotted hourly means as CSV. The file contains all 360 UTC hour-end timestamps from 20 September to 5 October 2026; blank cells preserve 37 missing hourly windows: 36 consecutive windows around the move and one earlier missing window. This is separate from the 37 high-temperature one-minute buckets above. Median source coverage was 240 scrapes per hour; seven available hours had fewer, with a minimum of 39. The observations before the long gap belong to AORUS; those after it belong to ASUS.
| Available hourly means | AORUS, before the long gap | ASUS, after the long gap |
|---|---|---|
| Paired observations | 278 | 45 |
| Power range, W | 17.6-277.1 | 21.3-275.7 |
| Temperature range, °C | 32.9-83.3 | 39.6-85.4 |
The ASUS maximum was 85.4 °C for the hour ending 4 October at 00:00 UTC, with mean power of 275.7 W, versus 83.3 °C for AORUS. That exceeds my withdrawn 85 °C cutoff in an hourly mean, not just a brief sampled peak; the workloads behind those maxima were not matched.
In the separate subset of one-minute points at 295 W or more, temperatures averaged 81.8 °C before the move and 81.2 °C afterward: 788 points before, 140 after. Fan readings averaged about 58% and 56%, respectively. Temperatures were similar within that high-power subset.
The thermal-violation series showed no increase, including during busy periods, but it has gaps and no captured positive event to validate that signal. Room temperature was not measured, and I have no host sensor series for the CPU, board or drives in the new enclosure. The measurements provide a baseline, without establishing whether the new enclosure improves cooling.
How the build changed#
| Configuration | What the evidence covers |
|---|---|
| Separate hosts: 96 GB on AORUS, 24 GB on its former host | Model comparisons; Gen5 x16 and Gen4 x8 link observations, respectively. |
| Consolidated ASUS host, two worker VMs | Gen5 x8/x8 observations, one weight-loading start and a cooling baseline. |
Before tuning inference further, I need to reconcile configuration drift, upgrade handling and alert formulas. Open platform checks include the firmware state behind the boot failure, host-memory headroom, transfer capacity under load, native IOMMU grouping without the inherited ACS override, and controlled cooling measurements.
Complete GPU onboarding checklist#
For a similar GPU build, work through these layers in order. Each step should end with an observation you can repeat, from powering on the host to answering a request after maintenance.
- Identify the exact card and platform. Record the GPU edition, power budget, slot wiring, enclosure and cooling. Follow the GPU manufacturer’s power-connection instructions; for this Max-Q, NVIDIA specifies its included adapter with two separate PCIe 8-pin PSU cables. Check any motherboard slot-power input. Include the NIC and read the board’s dual-card slot table.
- Check firmware and the physical link. Record the BIOS version and the VT-d, Above 4G and ReBAR states separately, along with the assigned BAR sizes. Locate failures at the host, VM, driver or engine layer. Compare capabilities with negotiated speed and width under load, including the upstream port.
- Prove the assignment boundary. Inspect IOMMU groups and all device functions; verify host binding and guest assignment. Check for ACS overrides before claiming native isolation. Keep one authoritative VFIO configuration and decide which GPU belongs to each VM.
- Bring up the guest runtime. Compare loaded-module and installed-driver versions, validate the GPU after reboot, and exercise a fresh workload through the NVIDIA container runtime.
- Verify scheduling. Check the selected device-plugin profile, allocatable count and rendered GPU request. Confirm that node labels, taints and RuntimeClass select the intended worker. Choose whole-card, time-sliced or MIG allocation deliberately, verifying support in the actual guest configuration. Advertised slots are not extra VRAM.
- Fetch one exact checkpoint. Measure disk fit, downloader memory and the real transfer path. Choose and mount the model disk, leave working space and avoid unused quantizations. If egress is restricted, test both allowed and denied destinations.
- Start the engine early. Use the intended image and checkpoint. Check KV-cache capacity, selected kernels, readiness and one representative tool call before a long comparison.
- Qualify the telemetry. Verify scrape coverage, per-card identity, units and missing-series behavior. Investigate impossible percentages before using the graph to choose a model.
- Promote the whole service. Move aliases, probes, replica policy and alerts together. Exercise the client-facing route, then repeat it after the planned restart or maintenance event. Check VM autostart policy, fresh-container driver compatibility and the rollback source.
- Keep the comparison reproducible. Save the input, engine and checkpoint identities, sample count, timing definition, power conditions and topology. Re-measure after hardware changes.



