RTX PRO 5000 72 GB + 48 GB: Local LLM Benchmark

I tested llama.cpp on my RTX PRO 5000 pair: one 72 GB card and one 48 GB card on a Z890 system (120 GB combined VRAM). The results compare single-card runs, layer splits, and tensor splits for the models listed below, all measured with the same settings.

Both cards are CPU-attached on separate root ports; the inter-GPU route is a PHB hop, no NVLink. Peer-path details and failure modes: GPU-to-GPU Bandwidth on Z890 and the fix guide.

All numbers below are under the 250 W cap (nvidia-smi -pl 250); stock 300 W will measure faster.

Measurement conditions

• Dates: matrix 2026-09-14/15; every cell from a stored run JSON with its command line and logs (the † paired cells: verbatim llama-bench output in the repo notes). The two § offload cells were measured 2026-09-20, on a newer checkout than this page’s pinned build — see Table notes. Flash attention on (fa: 1) for all recorded runs.
• Kernel booted intel_iommu=off iommu=off. Tensor split requires a healthy P2P path: if multi-GPU output is token soup, fix the platform first (guide) — nvidia-smi topo -p2p reports OK even when DMA is broken; a clean kernel log and stable output are the real evidence.
• GGML_CUDA_P2P=1 is not required post-fix; it was exported for most runs for protocol consistency and results are identical without it.
• Repetitions: -r 5 unless noted. Noise envelope: tg reproduces within ~1% across sessions; pp does not. Expect pp drift up to ~10% between sessions, and pp512 on big multi-GPU models swings hard even inside one session (five -r 5 samples of 122B tensor 3/2 read 1208–1669 in a single run). Compare pp only within one session; quote tg across sessions.

Results

Table notes

• Placement notation. 1×72 = whole model on one card (the 72 GB card, unless stated otherwise); tensor a/b / layer a/b = -sm tensor|layer -ts a/b, the GPU0/GPU1 weight ratio — bare -sm tensor is 1/1; layer auto = llama.cpp’s own layer balancer. On this pair the balancer’s allocation came out at roughly 3/2: on both 122B and Laguna, layer auto and explicit -ts 3/2 agreed within 0.2%.
• The same -ts numbers do not mean the same thing in both modes. In layer mode they weight whole-layer assignment — a layer goes to this card or that one; in tensor mode they slice the weights inside every tensor, so each card holds part of everything. The flag looks identical and the layout is not, which is why the § rows carry different -ts and -ncmoe values per mode.
• Repetitions. Cells are means of 5 repetitions (-r 5) except the rows marked † and §: those are single runs. The † pairs were measured back-to-back in one session at matched -ts 3/2, so each † pair is directly comparable to its partner. The § cells are single runs at the settings each mode needs; the § note below says how those settings were arrived at.
• Bold tg marks the fastest placement for that file on this box. It is split guidance for these weights, not a ranking of the models.
• ‡ = this llama.cpp build has no tensor mode for that architecture, so no tensor row exists: b10941 refuses Nemotron 3 Super with LLAMA_SPLIT_MODE_TENSOR not implemented for architecture 'nemotron_h_moe', and Flash Next the same way for qwen4exp. Those rows are layer placement on the same build — feature gaps in the build, not hardware problems. On Nemotron, layer auto and a biased layer split tied (76.5 vs 76.4 t/s).
• § — weights partly in system RAM. DeepSeek V4 Flash 0731 MXFP4 is 145.63 GiB against 118.39 GiB of VRAM on both cards (72,830 + 48,403 MiB), so running it means putting some experts in system RAM: -ncmoe keeps “the Mixture of Experts (MoE) weights of the first N layers in the CPU” — that flag’s own help text. From there the settings were adjusted by hand, mode by mode, down to the smallest offload that fits. In this file the expert tensors are 3.19 GiB per layer (43 layers, 256 routed experts, 6 used plus 1 shared per token), so -ncmoe 11 leaves ≈ 110.6 GiB of weights on the cards and -ncmoe 12 ≈ 107.4 GiB; whatever stays in RAM is read over PCIe on every generated token. Layer split fits at 4.5/2 with -ncmoe 11, tensor split at a flatter 3/2 with one more layer in RAM. Within those settings layer wins both phases — pp8192 217.3 vs 54.7 t/s, about ≈ 38 s versus ≈ 150 s to read an 8k prompt, and 19.4 vs 17.7 t/s at tg128 — the reverse of every resident file in this table. The figures belong to this offload layout, so they sit apart from the resident rows above. llama-bench labels the architecture deepseek4 ?B, so there is no active-parameter count for the Params column; sizes and the layer and expert counts come from the GGUF’s own tensor table. Measured 2026-09-20 on this tree’s CUDA build, which reports b11063 (3d82ef62d) — also where -ncmoe and the availability of tensor split for deepseek4 were read from source.
• Which rows share a session. The three Qwen3.8 27B rows and the Qwen3.6-35B-A3B 1×72 row are from the 2026-09-14 session. Muse Glimmer 30B and Gemma 4 31B 1×72 + tensor 1/1 rows, and the Qwen3.6-35B-A3B split rows, are from 2026-09-15 sessions; those models’ layer 1/1 rows come from a second 2026-09-15 session that also re-ran their tensor 1/1 splits, so the layer-versus-tensor comparison above is same-session for all three dense models. Rows from different sessions are still the same build, hardware and settings — tg reproduces within ~1%, pp much less (see Measurement conditions).
• Capacity versus throughput. Muse Glimmer 30B and Gemma 4 31B each fit one card outright (27.6 / 30.4 GiB), so their tensor 1/1 gains are throughput, not capacity. The ~95–101 GiB files are the opposite case: an even split would ask the 48 GB card to hold 47.4–50.7 GiB of weights before KV cache, compute buffers and driver overhead, which is why those rows are 3/2.

Laguna S 2.1 placement notes

• Capacity dictates the ratio: the 94.8 GiB Q6_K_L cannot halve onto the 48 GB card, so 3/2 (56.9 + 37.9 GiB) is the natural split — and the default layer balancer empirically lands at ~3/2 on this pair anyway (auto and explicit 3/2 agreed within 0.2% on both 122B and Laguna).
• Read the loader’s per-device buffer lines on asymmetric cards: they are what separates an allocation problem from a fault. Earlier quants of this model hit a balancer quirk that looked like a hardware failure; on Q6_K_L the buffers are clean.
• llama-bench labels -Q6_K_L shards as plain Q6_K (laguna 118B.A8B Q6_K, 94.83 GiB / 117.56 B params). Quote filenames, not model_type.
• Laguna ships as three GGUF shards; point -m at shard 00001 and the rest load automatically.

Commands

llama.cpp b10941 (4a8993735) for the table, except the two § DeepSeek V4 Flash rows, run from a newer checkout whose build reports b11063 (3d82ef62d); adjust paths to your tree.

# build
cmake -B build-cuda -DGGML_CUDA=ON && cmake --build build-cuda -j
BIN=~/src/llama.cpp/build-cuda/bin

# single-GPU control (repeat per card with --device CUDA0 / --device CUDA1)
$BIN/llama-bench -m ~/models/Qwen3.8/Qwen3.8-27B-Q8_0.gguf \
  --device CUDA0 -sm none -p 512,2048,8192 -n 128

# matched-ratio pair (same session; auto layer balancer picks ~3/2 on this box anyway)
$BIN/llama-bench -m ~/models/Qwen3.5/Qwen_Qwen3.5-122B-A10B-Q6_K_L/*.gguf \
  -sm layer -ts 3/2 -p 512,2048,8192 -n 128
$BIN/llama-bench -m ~/models/Qwen3.5/Qwen_Qwen3.5-122B-A10B-Q6_K_L/*.gguf \
  -sm tensor -ts 3/2 -p 512,2048,8192 -n 128

# Laguna matched pair (same session; run back-to-back)
$BIN/llama-bench -m ~/models/Laguna/Laguna-S-2.1-Q6_K_L/*00001-of-00003.gguf \
  -sm tensor -ts 3/2 -p 512,2048,8192 -n 128
$BIN/llama-bench -m ~/models/Laguna/Laguna-S-2.1-Q6_K_L/*00001-of-00003.gguf \
  -sm layer -ts 3/2 -p 512,2048,8192 -n 128

# DeepSeek V4 Flash MXFP4: bigger than both cards, so the first N layers'
# experts stay in system RAM (-ncmoe). Single repetition each.
$BIN/llama-bench -m ~/models/DeepSeek-V4-Flash-0731-MXFP4/*00001-of-00004.gguf \
  -sm layer -ts 4.5/2 -ncmoe 11 -r 1 -p 512,2048,8192 -n 128
$BIN/llama-bench -m ~/models/DeepSeek-V4-Flash-0731-MXFP4/*00001-of-00004.gguf \
  -sm tensor -ts 3/2 -ncmoe 12 -r 1 -p 512,2048,8192 -n 128

Common flags for the table cells: -ngl 999 -b 2048 -ub 512 -fa on -ctk f16 -ctv f16.

Two syntax notes, because they differ between programs:

• llama-bench separates devices inside one split with / (or ;); a comma starts a separate benchmark variant, so -ts 1,1 runs two single-GPU benches rather than one split run.
• llama-server / llama-cli accept either character as the device separator, so the same model placed here with tensor 3/2 runs as:

llama-server -m model.gguf --split-mode tensor --tensor-split 3,2 \
  -ngl 999 -fa on -ctk f16 -ctv f16

Details and the source check behind both: llama.cpp Common Options.