Hardware

PCIe 5.0 x8/x8 Isn't Enough: Testing GPU P2P Bandwidth on Z890

Two CPU-attached Gen5 x8 GPUs that move only 5.3 GB/s GPU-to-GPU in one direction and 2.0 GB/s in the other — measured with a custom probe and NVIDIA's p2pBandwidthLatencyTest, and why PCIe generation and width alone mislead.

Updated August 26, 2026Tested with NVIDIA RTX PRO 5000 Blackwell 72 GB (GPU0) · Intel Core Ultra 5 250K Plus (Arrow Lake) · NVIDIA 595.84 · Ubuntu 26.04 LTS, kernel 7.0.0-30-generic · NVIDIA RTX PRO 5000 Blackwell 48 GB · Intel Z890 — both GPUs on separate CPU root ports · PHB between the GPUs, both on NUMA node 0 · NVIDIA 595.84 · 13.2 runtime per nvidia-smi; 13.3 toolkit for the custom probe · NVIDIA cuda-samples p2pBandwidthLatencyTest · custom single-file CUDA DtoD benchmark (source in this article)

Two NVIDIA RTX PRO 5000 Blackwell GPUs, both attached straight to the CPU on a Z890 board, both negotiating PCIe 5.0 x8 under load, CUDA peer access enabled in both directions. A reasonable expectation for GPU-to-GPU transfer: most of the ~16 GB/s that a Gen5 x8 link should deliver.

The measured answer is 5.3 GB/s one way and 2.0 GB/s the other — and disabling P2P (letting transfers stage through system memory) was faster for bulk copies.

Test system

  • Intel Arrow Lake (Core Ultra 5 250K Plus) on a Z890 motherboard
  • GPU0: RTX PRO 5000 72 GB Blackwell, GPU1: RTX PRO 5000 48 GB Blackwell
  • NVIDIA driver 595.84; no LLM server or other CUDA workload running

Both cards are directly CPU-attached (separate root ports, not chipset) and on the same NUMA node:

        GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PHB     0-17            0               N/A
GPU1    PHB      X     0-17            0               N/A
00:06.0 -> 02:00.0 -> GPU0
00:06.3 -> 03:00.0 -> GPU1

One detail worth calling out because it rules out the obvious suspects: the cards idle at PCIe Gen1 x8, but under benchmark load both ramp correctly to Gen5 x8:

index, pstate, pcie.link.gen.current, pcie.link.width.current
0, P8, 5, 8
1, P8, 5, 8

So this is not a stuck-at-Gen3 link or a halved-width slot. The physical links are doing exactly what the spec sheet says.

Custom probe result

The first measurement used a small single-file CUDA program: enable peer access both directions, then time device-to-device cudaMemcpy transfers (0.5 GB and 1.1 GB) between the two cards’ buffers, plus a tight loop of 32-byte peer copies for latency. Run with no LLM server up:

GPU0 -> GPU1 (peer)      0.5 GB in 103.00 ms avg ->   5.21 GB/s
GPU1 -> GPU0 (peer)      0.5 GB in 276.99 ms avg ->   1.94 GB/s
GPU0 -> GPU1 repeat      0.5 GB in 102.61 ms avg ->   5.23 GB/s
32B copy x2048 avg latency: 4.03 us

GPU0 -> GPU1 (peer)      1.1 GB in 204.99 ms avg ->   5.24 GB/s
GPU1 -> GPU0 (peer)      1.1 GB in 566.05 ms avg ->   1.90 GB/s
GPU0 -> GPU1 repeat      1.1 GB in 207.08 ms avg ->   5.19 GB/s
32B copy x2048 avg latency: 4.02 us

Two things stand out: the ~2.7× directional asymmetry, and how repeatable it is — same numbers across buffer sizes and repeat runs, which makes background interference or clock lottery unlikely.

NVIDIA’s own sample confirms it (and adds the surprise)

NVIDIA’s p2pBandwidthLatencyTest from the CUDA samples repository almost exactly reproduces the asymmetry — and reports both P2P-enabled and P2P-disabled paths:

Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1
     0 1112.57  19.57
     1  14.35 1138.87

Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1
     0 1119.27   5.34
     1   2.05 1137.19

Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1
     0 1113.25  18.45
     1  18.38 1119.63

Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1
     0 1114.04   3.85
     1   3.89 1122.85

The surprise is in the disabled rows: with P2P off, copies stage through host memory and reach 19.57 GB/s (GPU0→GPU1) and 14.35 GB/s (GPU1→GPU0) — 3.7× and 7.3× faster than the “direct” peer path. Enabling P2P makes bulk transfers slower on this machine.

The latency tradeoff

P2P is not useless here — its latency is dramatically better:

P2P=Disabled Latency Matrix (us)      P2P=Enabled Latency (us)
GPU0 -> GPU1:  ~14.39                 GPU0 -> GPU1:  ~0.39
GPU1 -> GPU0:  ~14.38                 GPU1 -> GPU0:  ~0.38

So the peer path is functional and optimized for exactly one thing: very fast small transfers. It just does not move bulk data well on this platform.

What this probably means

Nothing in the measurements points at a defective board or card. The links negotiate correctly, both endpoints are CPU-attached, everything sits on one NUMA node, and results are repeatable to within a percent.

The limiting factor appears to be the peer-transaction path through the CPU root complex — the topology label nvidia-smi topo -m reports between the two GPUs:

  • PIX — at most one PCIe bridge
  • PXB — multiple bridges, no host bridge
  • PHB — crosses a PCIe host bridge / CPU root complex ← this system
  • SYS — additionally crosses the NUMA interconnect

A consumer platform can (and does) expose two fast, healthy PCIe links without providing equally fast forwarding between them. “PCIe 5.0 x8/x8” describes two point-to-point links; it says nothing about the route between their far ends. The host-staged path is faster for bulk transfers precisely because each half of it uses the link at its own healthy rate, with the CPU doing the relay.

What this means for local LLMs

The practical impact depends on which multi-GPU method you run:

  • Layer-split inference (llama.cpp’s default when a model is spread across cards) barely touches this path. Each token’s activation crosses once per layer boundary — tens of KB, single-digit microseconds even at 1.9 GB/s, against a 13–27 ms decode step. In our dual RTX PRO 5000 llama.cpp runs, splitting models across both cards cost essentially nothing versus the single-card equivalent per token, consistent with this being irrelevant at that payload size.
  • Tensor parallelism and other communication-heavy methods exchange data between GPUs on every layer of every token. Those are exactly the workloads where a 2–5 GB/s peer ceiling (and its asymmetry) starts to show. A “Gen5 x8/x8” build can look excellent on paper and be far from it in practice for these.

In short: judge a multi-GPU AI box by the measured peer path, not by the link labels.

Practical takeaway

Before buying or trusting a dual-GPU platform, measure the actual GPU-to-GPU route with the GPU idle (no LLM server running):

# 1. Topology: PIX/PXB/PHB/SYS between the GPUs, NUMA nodes
nvidia-smi topo -m

# 2. Negotiated links while a transfer is in flight (should hold Gen5 x8)
watch -n 0.2 'nvidia-smi --query-gpu=index,pstate,pcie.link.gen.current,pcie.link.width.current --format=csv'

# 3. Actual P2P bandwidth + latency (build once from the CUDA samples repo)
git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples && mkdir -p build && cd build
cmake .. && cmake --build . -j
./cpp/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest

The key numbers to compare are the P2P-enabled vs P2P-disabled bandwidth matrices (plus the latency matrices). If enabled-P2P is slower than host-staged on your box, you are on a platform with the same client-root-complex limitation — plan workloads around that.

Conclusion

These results are not evidence of a defective motherboard or GPU. The board is doing what it advertises: two CPU-attached PCIe 5.0 x8 links, both healthy, both ramping under load. What it does not advertise is how slow the route between those links is — 5.3 GB/s one way, 2.0 GB/s the other, with the peer path winning only on latency.

For multi-GPU local AI builds this is easy to miss from specification sheets alone: the interesting number is not what each link can do, but what the platform does when one GPU has to talk to the other. See also checking your GPU’s real PCIe bandwidth for the link-side diagnostics.

Raw results (full capture)
System:
GPU0: NVIDIA RTX PRO 5000 72GB Blackwell
GPU1: NVIDIA RTX PRO 5000 Blackwell 48GB
Driver: 595.84
CUDA reported by nvidia-smi: 13.2
Topology: PHB
NUMA: 0 for both GPUs

nvidia-smi topo -m:
        GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PHB     0-17    0               N/A
GPU1    PHB      X     0-17    0               N/A

PCIe capability / idle state:
GPU0 LnkCap: Speed 32GT/s, Width x16
GPU0 LnkSta: Speed 2.5GT/s, Width x8
GPU1 LnkCap: Speed 32GT/s, Width x16
GPU1 LnkSta: Speed 2.5GT/s, Width x8

Under benchmark load:
index, pstate, pcie.link.gen.current, pcie.link.width.current
0, P8, 5, 8
1, P8, 5, 8

Custom benchmark:
GPU0 -> GPU1 (peer)      0.5 GB in 103.00 ms avg ->   5.21 GB/s
GPU1 -> GPU0 (peer)      0.5 GB in 276.99 ms avg ->   1.94 GB/s
GPU0 -> GPU1 repeat      0.5 GB in 102.61 ms avg ->   5.23 GB/s
32B copy x2048 avg latency: 4.03 us

GPU0 -> GPU1 (peer)      1.1 GB in 204.99 ms avg ->   5.24 GB/s
GPU1 -> GPU0 (peer)      1.1 GB in 566.05 ms avg ->   1.90 GB/s
GPU0 -> GPU1 repeat      1.1 GB in 207.08 ms avg ->   5.19 GB/s
32B copy x2048 avg latency: 4.02 us

NVIDIA p2pBandwidthLatencyTest:
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1
     0 1112.57  19.57
     1  14.35 1138.87

Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1
     0 1119.27   5.34
     1   2.05 1137.19

Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1
     0 1113.25  18.45
     1  18.38 1119.63

Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1
     0 1114.04   3.85
     1   3.89 1122.85

P2P=Disabled Latency Matrix (us)
GPU:
0 -> 1: 14.39
1 -> 0: 14.38

P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU:
0 -> 1: 0.39
1 -> 0: 0.38

Sources and upstream documentation