Check Your GPU's Real PCIe Bandwidth on Linux

A GPU that should run at PCIe Gen5 x16 sometimes negotiates a lower link — wrong slot, riser cable, power-saving quirk. The good news: the real negotiated link is visible in two seconds, and for most inference workloads the impact is smaller than people expect.

Reading the lspci output

Each PCIe device reports two lines:

• LnkCap: Speed 32GT/s, Width x16 — capabilities advertised by the device (what it can do).
• LnkSta: Speed 32GT/s, Width x16 — the current negotiated link. A suffix like (ok) means it matches capability; anything else is a downgrade.

Speed names and approximate unidirectional throughput:

(Encodings reduce raw GT/s; the table uses effective payload figures.)

When a downgrade actually matters

Model inference (token generation) is memory-bandwidth bound on the GPU’s own VRAM, not on PCIe. A Gen5→Gen3 downgrade on an otherwise healthy system will barely move token-per-second numbers for single-GPU inference. Where it does matter:

• Model load time. A 70 GB GGUF streaming over Gen3 takes roughly 4× longer than Gen5. If you swap models frequently, this is the pain.
• Multi-GPU pipelines with layer or tensor parallelism, where activations and KV shards cross the link continuously.
• CPU-offloaded layers (-ngl less than total) — every offloaded layer pays the PCIe penalty on every token.

Why links downgrade

Work through this list in order:

1. Slot wiring. Many motherboards wire only one slot at full width/speed; secondary slots may be x4 or Gen3 even on expensive boards. Check the board manual, not the box art.
2. Riser cables / enclosures. Cheap risers are often Gen3 x4 or x8. A Gen5 card in a Gen3 riser shows exactly that mismatch in LnkSta.
3. BIOS settings. Some boards cap the link speed globally, or run slots in a shared-lane topology when multiple GPUs are present (x16 splits to x8/x8).
4. Power / training instability. Rare: a flaky link retrains at lower speed. Reseat the card and cable; if downgrades appear under load but not at boot, heat or power delivery is suspect.

Confirm topology with:

lspci -tv | grep -i -A2 nvidia

Measuring real bandwidth

Negotiated rate is a ceiling; for a practical number, time a large host→device copy. The classic tool is bandwidthTest from NVIDIA’s cuda-samples repository — a timed pinned-memory cudaMemcpy in both directions, reported as effective GB/s. It was removed from the repo after tag v12.8, and the old source no longer compiles on CUDA 13+ toolkits (it references the deleted cudaDeviceProp::computeMode field). So instead of archaeology, here is a minimal equivalent that builds anywhere with nvcc:

// h2d_bw.cu — effective PCIe host<->device copy bandwidth (GB/s), pinned memory
#include <cuda_runtime.h>
#include <cstdio>
#include <cstdlib>

#define CK(x) do { cudaError_t e = (x); if (e != cudaSuccess) { \
  printf("CUDA error %s at line %d\n", cudaGetErrorString(e), __LINE__); exit(1); } } while (0)

int main() {
  const size_t bytes = 256u << 20;  // 256 MiB per copy
  void *h, *d;
  CK(cudaHostAlloc(&h, bytes, cudaHostAllocDefault));
  CK(cudaMalloc(&d, bytes));
  cudaEvent_t t0, t1;
  CK(cudaEventCreate(&t0));
  CK(cudaEventCreate(&t1));

  CK(cudaMemcpy(d, h, bytes, cudaMemcpyHostToDevice));  // warm-up
  const int iters = 10;
  float h2d = 0, d2h = 0;
  for (int i = 0; i < iters; i++) {
    float ms;
    CK(cudaEventRecord(t0));
    CK(cudaMemcpy(d, h, bytes, cudaMemcpyHostToDevice));
    CK(cudaEventRecord(t1));
    CK(cudaEventSynchronize(t1));
    CK(cudaEventElapsedTime(&ms, t0, t1));
    h2d += (float)bytes / 1e9f / (ms / 1e3f);

    CK(cudaEventRecord(t0));
    CK(cudaMemcpy(h, d, bytes, cudaMemcpyDeviceToHost));
    CK(cudaEventRecord(t1));
    CK(cudaEventSynchronize(t1));
    CK(cudaEventElapsedTime(&ms, t0, t1));
    d2h += (float)bytes / 1e9f / (ms / 1e3f);
  }
  printf("Host -> Device: %6.2f GB/s\n", h2d / iters);
  printf("Device -> Host: %6.2f GB/s\n", d2h / iters);
  return 0;
}

# Save h2d_bw.cu (from the block above) in this working directory, then:
mkdir -p ~/pcie-bench && cd ~/pcie-bench
nvcc -O2 -arch=native h2d_bw.cu -o h2d_bw
./h2d_bw

Measured on this site’s RTX PRO 5000 test box (Gen5 x8, single copy engine busy, nothing else running):

Host -> Device:  28.75 GB/s
Device -> Host:  22.09 GB/s

A healthy PCIe link should reach a substantial fraction of its theoretical bandwidth, but the exact percentage varies by platform, transfer direction, pinned-memory behavior, NUMA topology, and benchmark implementation. Use the theoretical PCIe rate as an upper bound, not a strict expected result.

That measures host↔device speed over one link. A related but distinct number is GPU↔GPU P2P bandwidth — and it is the one that led us badly wrong here. On a Z890 box with two CPU-attached Gen5 x8 RTX PRO 5000s the peer path measured ~5.3 GB/s in one direction and ~2.0 GB/s in the other, slower than host-staged copies, while nvidia-smi topo -p2p reported OK in both directions. Those numbers were taken while Intel VT-d was rejecting DMA writes from GPU 0 (DMAR: [DMA Write NO_PASID]), which was also corrupting multi-GPU inference output; the link-side checks on this page were clean because the link was not the problem. The corrected write-up keeps the old figures as a faulty-IOMMU record, and Fix NVIDIA Multi-GPU P2P Corruption on Intel Linux is the diagnostic to run if your multi-GPU output is nonsense while the links look healthy.

Sources and upstream documentation

• lspci(8) — PCI enumeration and the LnkCap/LnkSta fields this page reads.
• setpci(8) — register-level access for the link-state checks.