An eGPU over Thunderbolt is not a plug-and-play PCIe slot. On this specific setup (AMD Strix Halo host + Sonnet 850T5 + RTX PRO 5000 Blackwell under Linux) the proven workflow is: boot with the enclosure disconnected, hot-plug it after login, then run a startup script before any CUDA workload. Every workaround in that script exists because something in this chain misbehaves without it — this page explains what, and labels how each explanation is sourced (captured vs assumed).
Why the disconnect-then-hot-plug dance
The recovered workflow is explicit: boot with the enclosure disconnected,
then hot-plug it after login. On this machine that sequence is required,
not just conservative: a cold-plug test from a full power cycle (booting
with the enclosure connected) left the GPU absent from every tool — no
function in lspci, nothing in nvidia-smi (observed 2026-08-24; no kernel
logs were captured from that boot, so the root cause of the missed
enumeration is unrecorded). Whatever it is — Thunderbolt authorization
timing, BIOS ordering, or firmware behavior — the practical rule stands: if
you boot with the enclosure connected and the card isn’t there, power-cycle
with it disconnected and hot-plug after login.
GPU detection
lspci -nn | grep -Ei "10de|nvidia|vga|3d|display|audio"
sudo lspci -vv -s 0000:63:00.0 | grep -E "Region|Resizable|LnkCap|LnkSta|LnkCtl"On the tested machine this resolves to the GPU function 0000:63:00.0
(VGA, PCI ID 10de:2bb3) and its sibling audio function 0000:63:00.1
(PCI ID 10de:22e8). PCI addresses are machine-specific — they depend on
the host’s bridge and topology. Re-derive them from your own output; the
startup script takes them as environment overrides.
PCIe topology of the Sonnet 850T5 chain
Observed chain on aibox1 (recovered script targets, confirmed by the
machine’s live lspci and the Thunderbolt bridge tree 00:01.2-[61-c0]) —
read it as this machine’s shape, not a universal one:
| Node | Address | What it is |
|---|---|---|
| Host TB root port | 0000:00:01.2 | CPU/SoC Thunderbolt root (host side) |
| Bridge | 0000:61:00.0 | “Barlow Ridge” — the Sonnet’s TB5 upstream bridge |
| Downstream port | 0000:62:00.0 | PCIe downstream to the slot |
| GPU | 0000:63:00.0 | RTX PRO 5000, function 0 (graphics), 10de:2bb3 |
| GPU audio | 0000:63:00.1 | HDMI/DP audio endpoint, 10de:22e8 |
The host’s Thunderbolt stack is active and in authorization mode
(bolt.service running; boltd: authmode enabled, force-power unset).
BAR sizing on this path: 256 MiB — and why that’s fine
This is the detail most eGPU write-ups get wrong. With ReBAR not in effect on the aibox1/Sonnet path, the card enumerates in small-BAR mode:
| BAR | Size / attributes (observed) |
|---|---|
| Region 0 (BAR0) | 64 MiB non-prefetchable, at 0xac00_0000 |
| Region 1 (BAR1) | 256 MiB 64-bit prefetchable, at 0x58_0000_0000 |
| Region 3 (BAR3) | 32 MiB 64-bit prefetchable, at 0x58_1000_0000 |
| ReBAR capability | BAR1 current 256 MB; supported sizes through 64 GB |
And nvidia-smi agrees: BAR1 total 256 MiB (used 1 MiB at the snapshot).
Despite the small aperture, CUDA sees and uses the full framebuffer — the
driver maps the rest behind the BAR. Practical consequence: on this path you
do not need big/Resizable BAR working; a 256 MiB BAR1 is the state the known
good configuration runs in. If your setup instead negotiates a full-size
BAR1, that’s different (and usually more) memory exposure — check
lspci -vv | grep -E "Region|Resizable" and nvidia-smi -q | grep -A8 "BAR1 Memory Usage" after hot-plug.
The workarounds, one by one
Full script with per-block explanation: Linux eGPU startup script reference. “Source” labels what backs each rationale; no without-tests have been run on this chain since the script was recovered, so most are assumed in the honest sense.
| # | Workaround | Why it is needed (source) | Status |
|---|---|---|---|
| 1 | power/control = on for root port, both bridges, GPU and audio (five devices) | Keeps the whole TB5 path out of PCI runtime autosuspend while CUDA workloads run (author-stated, assumed) | In recovered script; no without-test recorded |
| 2 | setpci CAP_EXP+30.w=0020:0020 (Link Control 2, bit 5) on root port + both bridges + GPU | Disables problematic link-speed auto behavior while allowing the GPU to ramp under load; same workaround appears in RTX PRO 6000 eGPU reports (author-stated, assumed; cross-referenced) | In recovered script; no without-test recorded |
| 3 | nvidia-smi -pm 1 (persistence mode) | Keeps the driver initialized between workloads; reduces reinitialization and hot-plug sensitivity (author-stated, assumed) | In recovered script; no without-test recorded |
| 4 | nvidia-smi -pl 275 (power cap, env-overridable) | Deliberately below the card’s 300 W stock default class to keep the enclosure and transients conservative. The 300 W default is confirmed by nvidia-smi on a directly attached 48 GB card (captured; enclosure rationale author-stated) | In recovered script |
| 5 | Unbind the GPU audio function (conditional — only if a driver is bound) | Not needed for compute; can complicate suspend/hot-plug behavior (author-stated, assumed) | In recovered script; no without-test recorded |
BIOS / firmware settings
Most rows are labeled by observed outcome; two rows reflect settings read directly from the machine’s BIOS:
| Setting | Outcome on aibox1 (evidence) |
|---|---|
| Above 4G Decoding | Effective — 64-bit prefetchable BARs are placed above 4 GiB (BAR1 at 0x58_0000_0000). Menu state not recorded. |
| Resizable BAR | OFF in BIOS — consistent with the observed 256 MiB BAR1: the card enumerates small-BAR despite ReBAR support to 64 GB, and that is what works here. |
| Secure Boot | OFF in BIOS (the sibling host aibox2 runs with it enabled). Not shown to affect this setup either way — no with/without test was done. |
| VT-d / IOMMU | Passthrough — iommu=pt on the kernel command line; no isolation issues observed with this setup. |
| Thunderbolt security level | Authorization enabled — boltd: authmode enabled, force-power unset (recovered capture). |
| PCIe Gen / link speed | Not exposed as a firmware toggle; negotiated by the stack — LnkCap 32GT/s x16 observed. |
If you change any of these during troubleshooting, record which one broke or fixed eGPU — that is the only labeling a menu walkthrough could add.
What about the 72 GB card?
The same RTX PRO 5000 Blackwell family exists in a 72 GB configuration. A directly attached 72 GB card was baselined on a different machine (aibox2: Ubuntu 26.04, kernel 7.0.0-30, driver 595.84, internal PCIe x16 slots) — useful context, but it does not prove Sonnet eGPU behavior:
| Fact (aibox2, direct attach) | Value |
|---|---|
| 72 GB card detected and running | FB 73,415 MiB total; same driver family as the 48 GB card beside it |
| BAR1 aperture in that slot | 65,536 MiB (64 GB) — not 128 GB |
| Power data (72 GB) | current/requested 250 W idle; default 300 W; max 350 W |
| Kernel log | no NVRM BAR/resource allocation failures |
Two implications for the eGPU decision: (1) a working 72 GB configuration does not require a 128 GB BAR1 window — that assumption is weaker than it looks; and (2) if the 72 GB card in the Sonnet path also stays at 256 MiB BAR1 like the 48 GB card does, it should behave much closer to the known-good case than to full-ReBAR eGPU reports — a lower-risk swap. The decisive check remains BAR1 size after hot-plug on aibox1 itself.
Verification after startup
Run these after hot-plug + script, in order (the first two are the ones that catch real problems):
# 1) BDFs present after hotplug
lspci -nn | grep -Ei '10de|nvidia|vga|3d|display|audio'
# 2) BAR sizing + ReBAR current size (expect the small-BAR state above)
sudo lspci -vv -s 0000:63:00.0 | grep -Ei 'Region 1|Region 3|Resizable|BAR|LnkCap|LnkSta|LnkCtl'
nvidia-smi -q | grep -A8 -Ei 'FB Memory Usage|BAR1 Memory Usage'
# 3) Driver/device health and power cap applied
nvidia-smi
nvidia-smi -q | grep -A5 -Ei 'Driver Version|CUDA Version|Product Name|Power Readings|PCIe Generation|Link Width'
# 4) Kernel log: no resource or NVRM failures this boot
sudo journalctl -k -b --no-pager | grep -Ei 'nvrm|bar|resource|resize|realloc|no space|thunderbolt|usb4|pciehp|63:00|61:00|62:00'
# 5) Link under LOAD, not just idle (idle links downshift — 2.5GT/s x4 at idle is normal here)
watch -n 0.5 'sudo lspci -vv -s 63:00.0 | grep -Ei "LnkSta:"'Do step 5 while a workload is running: the captured snapshot showed
LnkCap 32GT/s x16 with LnkSta 2.5GT/s x4 at idle — an idle reading
alone tells you nothing about what the link does under load.
For the deeper bandwidth question — what does Thunderbolt 5 actually give an AI workload compared to internal PCIe — the per-machine checker is Check your GPU’s real PCIe bandwidth.
Practical cost: model loading is slow over the link
The single most useful expectation to set before building this — and one this setup actually delivered: with the eGPU in use, loading a model can take several minutes, while the same model on the host’s local GPU loads almost instantly (observed on this setup, 2026-08-24).
That is bandwidth, not malfunction. When the model runs on the eGPU its weight file has to cross the Thunderbolt link instead of living in local memory; tens of gigabytes at whatever effective rate the encapsulated link delivers is “minutes” by construction, while the same load on the Strix Halo is a copy inside unified memory and finishes in seconds. It repeats on every model swap and every server restart, so factor it into which machine you actually reach for.
Sources and upstream documentation
- nvidia-smi documentation — the device and link-state fields this page reads.
- NVIDIA CUDA documentation — runtime behavior on Linux for validating the card.
- bolt — Thunderbolt host software;
bolt.servicestate is part of the verification. - setpci(8) — the register-level link workaround in the startup script.