RTX PRO 5000 Blackwell — llama.cpp Benchmarks

Token throughput (TG) and prompt processing (PP) for common local-AI workloads on a dual RTX PRO 5000 Blackwell box running llama.cpp with full GPU offload. Single-GPU numbers (72 GB card) are the primary set; dual-GPU rows use --device CUDA0,CUDA1 --split-mode tensor --tensor-split 3,2 (3:2 share toward the larger card).

One caveat on inter-card traffic: on a Z890/Arrow Lake build with this same GPU pair, measured GPU-to-GPU P2P bandwidth was only ~5.3 GB/s one way and ~2.0 GB/s the other (measurement). Layer-wise and tensor splits move tens of KB per token across that path, which is why the split results here were unaffected — but communication-heavy methods (true tensor parallelism) would feel that ceiling.

Results — single GPU (72 GB)

Model	Quant	Context	VRAM used	PP (t/s)	TG (t/s)
Qwen3 8B	Q8_0	128K	11.9 GB	1,480	96
Llama 3.1 8B	Q4_K_M	64K	7.4 GB	1,610	104
DeepSeek-R1-Distill-Qwen-14B	Q5_K_M	32K	12.8 GB	940	71
Qwen3 32B	Q4_K_M	32K	21.6 GB	585	44
Qwen3 32B	Q3_K_M	64K	18.9 GB	512	41
Llama 3.1 70B	Q3_K_M	32K	44.8 GB	268	19

Results — dual GPU (72 + 48 GB, tensor split)

Model	Quant	Context	VRAM used	PP (t/s)	TG (t/s)
Qwen3 32B	Q6_K	32K	30.1 GB	735	52
Llama 3.1 70B	Q4_K_M	32K	58.9 GB	391	27
Llama 3.1 70B	Q4_K_M	128K	79.4 GB	336	24

Reading the table:

PP scales with both cards almost linearly (compute bound); TG improves modestly (memory-bandwidth bound, and split tensors pay a sync cost).
KV cache dominates the long-context rows — 128K on a 70B model uses more VRAM than the weights themselves. Plan with GPU Model Fit before assuming a context “fits”.

Test command

Every row is produced by llama-bench on a quiet system (no other GPU work, Xid errors checked in dmesg):

# Prompt processing: 512-token prompt, no generation
./build/bin/llama-bench -m model.gguf -p 512 -n 0 \
  --threads 16 -ngl 99 -c <ctx> \
  --device CUDA0

# Token generation: 1-token prompt, 1,024 tokens generated
./build/bin/llama-bench -m model.gguf -p 0 -n 1024 \
  --threads 16 -ngl 99 -c <ctx> \
  --device CUDA0

-c is the context length from the table row; --device pins the single GPU (check which index nvidia-smi -L shows for the 72 GB card). Dual-GPU rows instead use both cards with a 3:2 tensor split:

--device CUDA0,CUDA1 --split-mode tensor --tensor-split 3,2

Methodology

Runs: three per configuration, median reported. Warm-up run discarded.
Context is the value passed to --ctx-size; KV cache quantization was default (f16) for all rows so results match common out-of-the-box setups.
VRAM used is peak from nvidia-smi during generation, including KV.
Environment: see the tested-environment block above. GPU clocks were left at defaults; this box idles at low temps under these loads (no sustained downclocking observed).

Reproducing

Build llama.cpp per this guide — record the commit with git rev-parse HEAD and note it next to your numbers.
Download the same model file (check its SHA-256 against the release).
Run the two llama-bench commands above, three times each.
If your numbers are within ~5% of this table, you’ve reproduced the setup; differences usually trace to clocks/power limits, KV quantization, or a different llama.cpp revision.

Sources and upstream documentation

llama.cpp repository — CUDA backend build and llama-bench; this page is pinned to the recorded revision, not current master.
NVIDIA CUDA documentation — runtime behavior assumed by the setup.