ROCm vs Vulkan on AMD Strix Halo: llama.cpp Benchmarks

Methodology

Primary tool: llama-bench — one standardized harness for both backends, with repetitions built in (-r) so each figure is a mean ± stddev.

Constant between the ROCm and Vulkan runs:

• llama.cpp commit (two builds from the same commit): c060ca974 — confirmed via build_commit in every retained JSON, both backends
• Model + quantization per section below (one dense, one MoE)
• GPU offload: -ngl 999 (all layers)
• Batch / ubatch: -b 2048 -ub 512
• Flash Attention: -fa on for both backends
• Test sizes: prompt processing at 512, 2048, 8192 tokens; token generation of 128 tokens (tg128); combined workload pg2048,256
• Repetitions: 5 per test (-r 5)
• System power mode fixed for the whole session: Balanced (the machine’s session power setting during all runs)
• No background workload during runs

JSON output is retained per backend/model as the source of truth; the tables below are formatted from it. Stability (crashes, NaNs, driver resets across repetitions) is noted per row where observed.

Models

One dense and one MoE model; context coverage comes from the pp test sizes (512 / 2048 / 8192):

Results

Per-test throughput in tok/s (mean ± stddev over 5 repetitions), from llama-bench -o json. Tests: pp512 / pp2048 / pp8192 — prompt processing; tg128 — token generation; pg2048+256 — combined 2048-token prompt then 256 generated tokens.

Dense model: Qwen3.8-27B-Q8_0

MoE model: Qwen3.6-35B-A3B-MTP Q8_0

35.5 B total parameters with ~3 B active per token (A3B), Q8_0, same file for both backends (37,790,106,112 bytes). The MTP head in the filename is inert under llama-bench — no --spec-type was given, so these are plain-decode numbers.

Cells filled only from the real runs; nothing estimated.

Speculative decoding: MTP on a realistic coding workload

The llama-bench table above answers “which backend is faster raw” under synthetic loads. This section measures how this machine performs in a real deployment: llama-server running with the practical settings you would actually use day-to-day — the model’s chat template, reasoning on, 131K context, tuned sampling parameters — and then asks whether speculative decoding helps on top of that. The workload is SPEED-Bench’s coding category against that server. If you are choosing a backend for real use rather than for a micro-benchmark record, this section is the more representative of the two.

Dense model (Qwen3.8-27B)

Draft acceptance was similar on both backends (~55–56% of drafted tokens accepted); the difference is how each backend converts that speculation into wall-clock time — a 2.06× overall speedup on ROCm vs 1.74× on Vulkan. The baseline gap here was 0.3%; with MTP, ROCm leads by 18.8% — a meaningfully different picture from the raw tg128 tie above.

Same workload, MoE model (Qwen3.6-35B-A3B-MTP)

Same server settings, same coding workload, same spec flags — only -m changed.

Three things differ from the dense model’s story:

• At baseline, Vulkan leads generation — by +4.9% here (47.83 vs 45.58), same direction as the synthetic tg128 result (+12.5%). The lead is smaller on the realistic workload than on the synthetic one; different instrument, same ranking.
• With MTP, the backends converge. Both land at ~60 tok/s and ROCm’s lead shrinks to +0.9% (60.92 vs 60.40). On this model MTP is not a backend tiebreaker — it cuts latency (−23% / −20%) whichever backend you pick.
• Higher acceptance, smaller percentage gain. The MoE model accepts more drafts (70.5% / 72.6%) than the dense one (~55–56%), yet its MTP gain is much smaller in percentage terms: +33.6% / +26.3% vs +105.7% / +73.6%. A model already generating ~45–48 tok/s leaves less expensive serial work for speculation to eliminate — acceptance rate alone does not set the size of the speedup; the baseline cost per token does.

Cross-model MTP summary

The takeaway across both models: the value of speculative decoding depends heavily on the model’s baseline cost per token. A dense 27B at ~7.6 tok/s has room for a 2× gain; a 35B-A3B MoE already near 48 tok/s has much less — even though its MTP head is better (higher acceptance).

Other speculative method tested: ngram-map-k4v (negative result)

The tested configuration (below) found almost no repeated continuations in this coding workload on the dense model, and produced exactly zero draft tokens on both backends of the MoE model — nothing to speed up in either case. This establishes only that these settings on this workload were inactive across both models tested — not that ngram-map-k4v cannot help other workloads. The broader lesson: enabling a speculative method does not automatically buy a speedup; the method must actually generate accepted drafts.

How these runs were done

The exact tested command (ROCm build shown; the Vulkan side used the identical command with build-vulkan/bin/llama-server instead). The standard MTP option is the only variant worth spelling out in full:

./build-rocm/bin/llama-server \
  -m /home/user/models/Qwen3.8/Qwen3.8-27B-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.05 \
  --chat-template-file /home/user/models/Qwen3.8/chat_template.jinja \
  --reasoning on -ngl 999 -c 131072 --threads 14 --no-mmap --jinja \
  --chat-template-kwargs '{"reasoning_effort":"xhigh"}' \
  -np 1 -b 2048 -ub 512 -fa 1

The other eleven runs are variants of that one line:

• Baseline (×4): the same line with no --spec-* flags at all.
• ngram-map-k4v (×4, the negative-result runs): swap the two MTP flags for --spec-type ngram-map-k4v --spec-ngram-map-k4v-size-n 16 --spec-ngram-map-k4v-size-m 24 --spec-ngram-map-k4v-min-hits 2.
• MoE round (×4): identical to the dense round with only -m changed to /home/user/models/Qwen3.6/Qwen3.6-35B-A3B-MTP-Q8_0.gguf.

Workload (identical for all twelve runs — six per model):

cd ~/code/github/llama.cpp && source .venv-speedbench/bin/activate
python tools/server/bench/speed-bench/speed_bench.py \
  --url localhost:8080 --bench qualitative --category coding \
  --osl 512 --concurrency 1 --limit 8 --output <result-name>.json

• bench qualitative, category coding, concurrency 1, OSL 512, temperature 0 (supplied by the benchmark), 8 samples selected / 8 completed / 0 failed, 10 turns per sample.
• The server’s -c 131072 is a realistic deployment setting; this workload is not a large-prompt test, so no large-context claims come from this section.
• --reasoning on: Qwen3.8 is a thinking model — the reported tok/s include reasoning tokens.

Caveats before reading into these numbers

• Do not compare SPEED-Bench prompt-throughput across backends or runs. The retained JSONs show wide run-to-run spread in aggregate prompt throughput (≈57–208 tok/s across the six dense-model files, 302–526 across the six MoE files); generation throughput is stable. Backend prompt-processing claims come from the llama-bench pp cells above — the cleaner instrument for that question.
• Earlier throughput_32k-labeled runs are deliberately excluded: their recorded requests contained only 68 prompt tokens — not a genuine 32K test.
• The baselines here sit near the synthetic tg128 cells with instrument differences: dense 7.62 / 7.60 vs 7.76 / 7.77 (same ranking — a tie); MoE 45.58 / 47.83 vs 47.47 / 54.22, where Vulkan’s lead is narrower on the realistic workload (+4.9%) than synthetic (+12.5%). Same ranking in both cases — different instruments, not contradictions. Real prompts, long generations including reasoning tokens.
• Reproducibility: model files as in the Models table (29,105,393,664 bytes dense / 37,790,106,112 bytes MoE), ROCm 7.14 / Mesa RADV 26.1.7, -ngl 999 -b 2048 -ub 512 -fa 1, server context 131072, flags as shown. llama.cpp: c060ca974 (build 10603) — recorded in every retained llama-bench JSON (both models, both backends). All twelve server runs used the same checkout and server configuration except for the variables being tested.
• Don’t over-generalize the architecture comparison. What is established: in this test, Qwen3.6-35B-A3B MoE generated tokens far faster than Qwen3.8-27B dense under the same methodology (~45–48 vs ~7.6 tok/s). “MoE is always faster than dense” does not follow from this data.

Interpretation

Findings cover two models on one machine (power profile Balanced), the same llama.cpp commit and byte-identical parameters: a dense 27B and a 35B-A3B MoE, both Q8_0. Other architectures, quants, and sizes may shift the results.

Dense model (Qwen3.8-27B)

• Prompt-processing winner: ROCm/HIP, by a consistent margin that holds at every tested size: +22.4% (pp512), +20.3% (pp2048), +19.4% (pp8192). The ranking does not change with prompt size — it narrows slightly as prompts get longer but never inverts.
• Token-generation: a statistical tie. 7.76 vs 7.77 tok/s is inside both backends’ stddev (≈0.1% apart) — for this model, generation speed gives no reason to prefer either backend.
• Combined workload: pg2048+256 goes to ROCm by +5.6%. The gap is much smaller than the pure pp numbers because the cell spends most of its wall time in generation, where the backends tie — mixed real use therefore lands between “tied” and “pp-dominated”.

MoE model (Qwen3.6-35B-A3B-MTP) — generation flips to Vulkan

• Token-generation goes to Vulkan by a clear margin: 54.22 vs 47.47 tok/s (+12.5%). The gap is far outside both backends’ tiny stddevs (±0.09 / ±0.07), and each backend’s p-run tg and pg-run tg re-runs agree with each other to under 0.4% — stable, not a fluke.
• Prompt processing narrows to near-parity. ROCm keeps a lead at every size — +3.9% (pp512), +3.8% (pp2048), +1.6% (pp8192) — but the dense model’s ~20% advantage largely does not carry over: at pp512 the gap is inside ROCm’s elevated run-to-run variance, and at 2048/8192 it is a real but small difference (~3 SE with 5 reps).
• Combined workload follows generation: Vulkan +6.9% on pg2048+256.
• Realistic-workload check (SPEED-Bench section): Vulkan’s generation lead narrows to +4.9% at baseline (47.83 vs 45.58 tok/s), and MTP closes it — both backends reach ~60 tok/s with ROCm +0.9%. On this model MTP buys latency (−20–23%), not a backend tiebreaker.
• Absolute speeds sanity-check the architecture: MoE prefill runs ~3× the dense model (≈1,000 vs ≈300–340 tok/s) and generation ~6–7× it (≈47–54 vs ≈7.8), consistent with ~3 B active weights instead of 27 B.
• No mechanism claim: we did not instrument why Vulkan wins MoE generation (expert dispatch is a many-small-GEMM pattern; which backend handles it better was not measured). The result stands as observed.

Cross-model observations

• Memory efficiency: not measured in this round — no startup “VRAM used” log lines were captured, so the tables above are throughput only.
• Stability: all five repetitions of every cell produced valid numbers on both backends — no crashes, NaNs, or driver resets observed. The elevated pp512 variance is symmetric (ROCm ≈3.6%, Vulkan ≈3.4%), so it is a property of the short-prompt test setup, not of one backend.
• Acceptance rate does not set the speedup. The MoE model accepts more drafts (~71–73%) than the dense one (~55–56%) yet gains less in percentage terms (+34 / +26% vs +106 / +74%): with a baseline already near 48 tok/s there is less expensive serial generation for speculation to remove. The ceiling on speculative decoding’s value is the model’s baseline cost per token.
• Installation complexity: ROCm 7.14 required more moving parts — AMD repo key + multi-arch repo line, a gfx1151-suffixed entry package, and a shell-environment block for the build (see Install ROCm 7.14 on Strix Halo); on this machine it also meant migrating off a prior 7.13 install, which is where most of the fiddliness sat. Vulkan was a PPA add plus apt packages with a one-flag cmake build — simpler in steady state, but you must know which Mesa source gives a current RADV for this hardware (see Set Up Vulkan Compute on Strix Halo). Neither required BIOS or kernel parameter changes here.

Practical recommendation

Pick by model type — on this hardware no single backend wins both families:

• Dense models → ROCm/HIP. ~20% faster prompt processing at every size, token generation tied, and with MTP speculative decoding (section above) it also takes the generation lead — 15.67 vs 13.19 tok/s, +18.8% on Qwen3.8-27B. For dense or thinking models this is the stronger stack end to end.
• MoE models → Vulkan at baseline; effectively tied once MTP is on. Without speculative decoding, Vulkan wins generation (54.2 vs 47.5 tok/s synthetic tg; +4.9% on the realistic coding workload) with prompt processing only ~2–4% behind, and it is the simpler install. With MTP enabled both backends converge to ~60 tok/s (ROCm 60.92 vs Vulkan 60.40, +0.9%) — past that point the choice comes down to ROCm’s small prompt-processing lead and how much you value the lighter stack.
• Mixed use → build both. Each build is one cmake flag apart on the same source tree — pick per model at runtime, or default to whichever family your workload is dominated by.

Scope: two models (one dense, one MoE), one quant (Q8_0), one machine, power profile Balanced. The pattern to trust most is that backend advantages on this hardware are model-shape-dependent — re-measure before generalizing to other architectures or sizes.

All eight retained JSON files record: build_commit c060ca974 (build 10603), the identical model file per pair (29,105,393,664 B dense / 37,790,106,112 B MoE), -ngl 999 -b 2048 -ub 512 -fa on, 5 repetitions each, Vulkan device Radeon 8060S Graphics (RADV STRIX_HALO) with devices: auto (no forced selection).

Reproduce

# ROCm build (build-rocm.sh at the repo root — see /guides/install-rocm-7-14-strix-halo/):
export HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)"
cmake -S . -B build-rocm -G Ninja \
  -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_HIP_ROCWMMA_FATTN=OFF \
  -DGGML_HIP_NO_VMM=ON -DGGML_HIP_MMQ_MFMA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-rocm -j"$(nproc)"

# Vulkan build (build-vulkan.sh at the repo root — see /guides/vulkan-compute-strix-halo/):
cmake -S . -B build-vulkan -G Ninja -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-vulkan -j"$(nproc)"

# Benchmark — the exact commands run on the tested machine:
MODEL="/home/user/models/Qwen3.8/Qwen3.8-27B-Q8_0.gguf"   # path on the tested machine — use your own GGUF
mkdir -p benchmark-results

# ROCm side:
./build-rocm/bin/llama-bench  \
  -m "$MODEL" -ngl 999 -p 512,2048,8192 -n 128 -b 2048 -ub 512 -fa on -r 5 -o json \
  > benchmark-results/rocm-7.14.json
./build-rocm/bin/llama-bench  \
  -m "$MODEL" -ngl 999 -pg 2048,256 -b 2048 -ub 512 -fa on -r 5 -o json \
  > benchmark-results/rocm-7.14-pg2048-256.json

# Vulkan side — byte-identical parameters, different build + output names:
./build-vulkan/bin/llama-bench  \
  -m "$MODEL" -ngl 999 -p 512,2048,8192 -n 128 -b 2048 -ub 512 -fa on -r 5 -o json \
  > benchmark-results/vulkan-26.1.7.json
./build-vulkan/bin/llama-bench  \
  -m "$MODEL" -ngl 999 -pg 2048,256 -b 2048 -ub 512 -fa on -r 5 -o json \
  > benchmark-results/vulkan-26.1.7-pg2048-256.json

# MoE round — byte-identical parameters, different GGUF (same four cells per
# backend):
MODEL="/home/user/models/Qwen3.6/Qwen3.6-35B-A3B-MTP-Q8_0.gguf"   # path on the tested machine — use your own GGUF
./build-rocm/bin/llama-bench  \
  -m "$MODEL" -ngl 999 -p 512,2048,8192 -n 128 -b 2048 -ub 512 -fa on -r 5 -o json \
  > benchmark-results/rocm-7.14-moe.json
./build-rocm/bin/llama-bench  \
  -m "$MODEL" -ngl 999 -pg 2048,256 -b 2048 -ub 512 -fa on -r 5 -o json \
  > benchmark-results/rocm-7.14-pg2048-256-moe.json
./build-vulkan/bin/llama-bench  \
  -m "$MODEL" -ngl 999 -p 512,2048,8192 -n 128 -b 2048 -ub 512 -fa on -r 5 -o json \
  > benchmark-results/vulkan-26.1.7-moe.json
./build-vulkan/bin/llama-bench  \
  -m "$MODEL" -ngl 999 -pg 2048,256 -b 2048 -ub 512 -fa on -r 5 -o json \
  > benchmark-results/vulkan-26.1.7-pg2048-256-moe.json

# Model files: both GGUFs are from bartowski on Hugging Face (file names as in
# the Models table above).

Raw data

The retained outputs behind every table on this page are available for inspection:

• ROCm vs Vulkan raw data (ZIP) — the eight llama-bench JSONs, the twelve SPEED-Bench JSONs, and a README with the environment, commands, and exclusions.

Raw benchmark output and exact commands are retained with this article. Results are specific to the tested hardware, models, quantizations, software versions, and runtime settings.

Sources and upstream documentation

• llama.cpp repository — upstream project; this page is pinned to commit c060ca974 (build 10603), not current master.
• llama.cpp server documentation — the llama-server flags used by the MTP round.
• SPEED-Bench — methodology behind the coding-workload round.
• Qwen on Hugging Face — official model repositories for both tested models.
• bartowski GGUFs on Hugging Face — source of the exact weight files used.