Fix NVIDIA Multi-GPU P2P Corruption on Intel Linux

Investigate corrupt multi-GPU llama.cpp output using kernel logs and tests for IOMMU-related problems.

First published September 14, 2026Updated September 16, 2026

System details: RTX PRO 5000 workstation (Z890)

My dual RTX PRO 5000 setup produced corrupt output during multi-GPU inference: sentences that dissolved into random tokens, and eventually a cuBLAS failure deep into a long context. On this Z890 system the cause was the IOMMU rejecting DMA writes from the GPU, and disabling IOMMU translation resolved it. These are the checks I used to investigate.

Use the diagnostic steps to determine whether your system has the same problem before changing its IOMMU settings. This is a procedure for finding that out, not a recommendation to turn off DMA isolation everywhere.

Symptom checklist

Check for IOMMU issues when several of these occur together:

  • Output from a multi-GPU llama.cpp / ik_llama.cpp run looks like shuffled multilingual fragments rather than text (Glob headoinedeliverjudT head hint JP…), while the same model on one GPU is coherent.
  • The failure is not a clean crash: it produces confident nonsense first, and/or dies later inside CUDA with something like CUDA error: an internal operation failed … in function ggml_cuda_op_mul_mat_cublas.
  • Running the same model through llama.cpp RPC works (two processes, transfers crossing a process boundary instead of a direct peer copy).
  • A llama.cpp build configured with -DGGML_CUDA_NO_PEER_COPY=ON — host-staged transfers — is stable.
  • dmesg is full of DMAR: [DMA Write NO_PASID] Request device [<BDF>] lines.
  • The machine is Intel (VT-d) with IOMMU enabled by default, and the GPUs are two separate PCI functions rather than an NVLink pair.

If only the first item is true, suspect your build or template before suspecting DMA. It is the combination — corrupt output on the direct peer path, clean output on a host-staged path — that points at the transport.

Why “P2P: OK” does not mean the peer path works

nvidia-smi topo -p2p r
nvidia-smi topo -p2p w
        GPU0    GPU1
GPU0      X      OK
GPU1     OK       X

That matrix reports capability: the driver believes peer access is supported and permitted. The machine, before the fix, reported the same OK in both directions while the kernel was busy rejecting the GPU’s DMA writes. nvidia-smi topo -p2p cannot see the IOMMU, and it also cannot tell you whether llama.cpp is actually using the peer path (in current llama.cpp, direct CUDA peer access is opt-in at runtime — see the option reference).

In short: this machine’s capability matrix reported peer access as available the whole time it was logging DMA faults. Check the application output and the kernel log as well as the capability report — capability is a statement about the link, correctness is about the transactions that cross it.

Step 1 — Read the kernel’s DMA fault log

sudo dmesg -T | grep -Ei 'DMAR|IOMMU|NO_PASID|NVRM|Xid|fault'

The line that matters:

DMAR: [DMA Write NO_PASID] Request device [02:00.0] fault addr 0xa000002000
      [fault reason 0x71] SM: Present bit in first-level paging entry is clear

DMA Write + NO_PASID means an untranslated device write (a GPU DMA engine) hit an address the first-level page tables do not map, so VT-d dropped it. A dropped write inside a tensor copy is exactly the “coherent model, corrupt output” signature — nothing reports an error to CUDA, the data is just wrong.

These are not discovery-time noise. On this box they kept arriving in clusters hours into sessions; the journal also carried suppression notices of the form dmar_fault: N callbacks suppressed, so what you see badly understates what happened. Count them over the whole retained journal to get the scale and the date range:

journalctl --no-pager | grep -ciE '\[DMA (Write|Read) NO_PASID\]'
journalctl --no-pager | grep -oiE 'Request device \[[0-9a-f:.]+\]' | sort | uniq -c
journalctl --no-pager | grep -oE 'dmar_fault: [0-9]+ callbacks suppressed' \
  | awk '{ if ($2 > m) m = $2 } END { print "largest suppression burst:", m }'

Output from the affected machine (whole journal, ~5 weeks):

2980                                     # logged faults, 2026-08-14 → 2026-09-13
   2916 Request device [02:00.0]         # the 72 GB card
     64 Request device [03:00.0]         # the 48 GB card
largest suppression burst: 3175687       # callbacks behind one "suppressed" notice

All 2,980 are DMA Write (never DMA Read) with the same reason code 0x71, and 2,916 of them name one device. A one-sided pattern like that — writes from a single function — is worth reporting to NVIDIA, but it did not need an answer before the box could be made reliable.

Watch them live while a run is in flight:

sudo journalctl -kf | grep --line-buffered -Ei 'DMAR|IOMMU|NO_PASID|NVRM|Xid|fault'

Note what you are looking at: Xid lines are NVIDIA driver errors and DMAR lines are Intel VT-d errors. Xid-free but DMAR-full is the fingerprint of an IOMMU problem rather than a GPU problem.

Step 2 — Map the faulting BDF to a GPU

lspci -Dnn | grep -i NVIDIA
lspci -nnk -s 02:00.0
nvidia-smi --query-gpu=index,name,memory.total,pci.bus_id --format=csv
0000:02:00.0 VGA compatible controller: NVIDIA Corporation GB202GL [RTX PRO 5000 Blackwell]
        Kernel driver in use: nvidia

0, NVIDIA RTX PRO 5000 72GB Blackwell, 73415 MiB, 00000000:02:00.0
1, NVIDIA RTX PRO 5000 Blackwell, 48935 MiB, 00000000:03:00.0

Write the index↔BDF mapping down before you go any further (the 02:00.0 / 03:00.0 addresses throughout this page are this machine’s — substitute your own). Asymmetric cards are easy to mislabel: this pair reports identical PCI device IDs even though one card has 72 GB and the other 48 GB — only nvidia-smi’s memory column tells them apart.

Also worth grabbing while you are here, because it frames everything downstream:

nvidia-smi topo -m      # PHB / PXB / PIX / SYS between the GPUs, NUMA nodes
cat /proc/cmdline       # does it already mention intel_iommu or iommu?

Step 3 — Test with host-staged copies

Before touching the kernel command line, build a llama.cpp that never issues a direct peer copy:

cmake -B build-cuda-nopc -DGGML_CUDA=ON -DGGML_CUDA_NO_PEER_COPY=ON
cmake --build build-cuda-nopc -j

GGML_CUDA_NO_PEER_COPY is a compile-time definition: the peer cudaMemcpyPeerAsync paths are compiled out and the caller falls back to staging through host memory (it also turns off the copy/compute-overlap capability, so expect some slowdown). If the garbage output disappears in that build while the same model, quant, seed and prompt still fail in a normal build, you have localized the fault to the direct device-to-device path rather than to the model or the quantization.

Do not confuse it with the runtime variable GGML_CUDA_NO_PINNED, which only turns off pinned host memory — that is a staging-buffer knob, not a P2P switch, and it is not a fix for a broken peer path.

On this machine the host-staged build was stable. (ik_llama.cpp has no usable equivalent control: its GGML_CUDA_NO_PEER_COPY builds eventually segfaulted in the CPU-side fallback path at the tested revision — specific to that build, not a general claim.) llama.cpp RPC worked too, for the same underlying reason: different process topology, different transfer path. These controls point at the transport; the DMAR log is what ties it to the platform’s DMA path.

Step 4 — Temporarily disable IOMMU translation

Test without permanently editing the bootloader: at GRUB, press e, add intel_iommu=off iommu=off to the line starting linux, then press Ctrl+X to boot. Rebooting back into the old entry reverts it.

If you want to keep the change across boots while you evaluate it, edit GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub and run sudo update-grub. Both parameters are belt-and-braces: intel_iommu=off turns the Intel driver off, iommu=off is the generic kernel switch.

Step 5 — Verify what actually booted

Three checks, in order of directness:

cat /proc/cmdline                                  # parameters present?
sudo dmesg | grep -Ei 'DMAR|IOMMU'                 # decisive line below
readlink /sys/bus/pci/devices/0000:02:00.0/iommu_group
readlink /sys/bus/pci/devices/0000:03:00.0/iommu_group
BOOT_IMAGE=/boot/vmlinuz-7.0.0-31-generic ro quiet splash intel_iommu=off iommu=off …
DMAR: IOMMU disabled
(no output from readlink — the iommu_group symlinks are gone)

Two things about that dmesg output worth knowing:

  1. DMAR lines after IOMMU disabled do not mean translation is still active. The kernel still parses the ACPI DMAR table and prints DRHD, dmar0/dmar1, SATC and DMAR-IR lines. Read for the one line DMAR: IOMMU disabled.
  2. IRQ remapping stays on — you will see DMAR-IR: Enabled IRQ remapping in x2apic mode. Interrupt remapping is a separate feature from DMA isolation; the empty /sys/class/iommu/ and the vanished per-device iommu_group links are what tell you device DMA translation is off.

Then re-confirm capability, and — the step most often skipped — that the faults stopped:

nvidia-smi topo -p2p r && nvidia-smi topo -p2p w
journalctl -k --since boot | grep -ciE '\[DMA (Write|Read) NO_PASID\]'   # expect 0

Step 6 — Verify application output

Speed numbers alone do not clear this bug, because the original failure showed up as wrong output, not slow output. For any split configuration you intend to trust:

  1. Start llama-server with a pinned build and fixed model; use greedy or otherwise deterministic sampling (--temp 0, fixed --seed).
  2. Run the same short prompt single-GPU and multi-GPU. Both must be coherent; byte-identical output is nice but not required, since backends can reorder floating-point work.
  3. Run one realistic heavy request (a coding-agent style prompt), then exercise a context length past wherever the old failure appeared — ours surfaced near 128K.
  4. Keep the journal watch from Step 1 open and confirm zero new DMAR/NVRM/Xid lines for the whole session.

Then measure. On this box (both cards power-capped at 250 W rather than the 300 W default), once the peer path was healthy, tensor-split scaling on a model that fits either card became large and obvious — the scripted -r 5 control matrix has Qwen 3.8 27B Q8_0 at 38.6 t/s generation on one GPU and 62.3 t/s across both at -sm tensor -ts 1/1 (+61%), with prompt processing fastest instead on layer 1/1 (3,635.7 t/s at pp4096 against 2,184.6 t/s single-card, +66%). A genuine 131,072-token prompt on the tensor split (llama-bench -p 131072, 2,098.5 t/s) — the shape that died inside cuBLAS before the fix — completed with no new kernel or CUDA events, and daily dual-GPU serving on this box produces coherent output across these configurations. The full matrix is in the RTX PRO 5000 benchmark page, and the before/after peer-bandwidth measurements — asymmetric 5.2/1.9 GB/s becoming a symmetric ~5.1, with the bulk ceiling itself unchanged — are in the P2P bandwidth benchmark.

# llama-bench uses '/' between devices; ',' separates benchmark variants.
GGML_CUDA_P2P=1 ./build-cuda/bin/llama-bench \
  -m /home/user/models/Qwen3.8/Qwen3.8-27B-Q8_0.gguf \
  -ngl 999 -sm tensor -ts 1/1 -ctk f16 -ctv f16 \
  -b 2048 -ub 512 -fa on -p 4096 -n 256 -r 5 -o json

Decide whether to keep the setting

Reasonable to keep, if all of these hold: the box is dedicated to inference, physically trusted, runs no VMs and no VFIO/passthrough devices, has no Thunderbolt or PCIe-enclosure peripherals you would ever trust less than your GPU, and you have a dated note on the machine saying why.

Go back to IOMMU-on and investigate properly if any of these hold: you need passthrough or VMs, the box is reachable by untrusted peripherals, or — most practically — a BIOS update, kernel version or NVIDIA driver branch fixes the fault without the tradeoff. Re-test after each firmware/kernel change; this class of bug can move.

To revert: remove the two parameters from /etc/default/grub, sudo update-grub, reboot, then confirm readlink /sys/bus/pci/devices/0000:02:00.0/iommu_group returns a path again.

Diagnostic checklist

  • sudo dmesg -T | grep -Ei 'DMAR|NO_PASID|Xid' — are writes being rejected, and by which BDF?
  • lspci -nnk -s <BDF> + nvidia-smi --query-gpu=index,name,memory.total,pci.bus_id — BDF→GPU mapping written down
  • Garbage output on direct peer path, coherent on host-staged (-DGGML_CUDA_NO_PEER_COPY=ON) or RPC?
  • Tested at GRUB edit-time (temporary) rather than editing the bootloader first?
  • DMAR: IOMMU disabled in dmesg, per-device iommu_group links gone
  • Zero new NO_PASID faults in the current boot’s journal under load
  • Coherent greedy output single- and multi-GPU, at long context, with a real prompt
  • Journal quiet for the whole session (DMAR, NVRM, Xid)
  • Decision recorded: keep or revert, and why — dated

Sources and upstream documentation

Spotted a broken command or a wrong number?Email me — corrections get fixed on the page.