Build llama.cpp with CUDA for NVIDIA Blackwell

Build llama.cpp for Blackwell GPUs on Linux, with the CUDA settings and commands I used.

Updated August 22, 2026

System details: RTX PRO 5000 workstation (Z890)

The prebuilt binaries of llama.cpp rarely include the right SASS for brand-new GPUs, and “just add CUDA” is not enough: Blackwell needs an explicit compute capability or the kernel image is silently not compiled in. This guide gets a working, GPU-accelerated build on Ubuntu with a current toolkit.

Prerequisites

  • GPU: NVIDIA Blackwell with a datacenter, workstation or gaming SKU (RTX PRO 5000/6000, RTX 50xx series). Compute capability sm_120 for workstation/consumer; sm_100 for B100/B200/GB200 class.
  • Driver: recent enough for your CUDA toolkit version — check with nvidia-smi, which prints the supported CUDA release in the top-right corner.
  • CUDA toolkit (optional but recommended): a system install of the cuda-toolkit package, or let CMake find one you point at via CUDA_HOME.
  • Build tools: CMake 3.24+, Ninja (or make), a C++17 compiler, git.

Install the CUDA toolkit

llama.cpp needs nvcc from a toolkit version that knows about your architecture. On Ubuntu:

# Add the NVIDIA repository (skip if you already have CUDA installed)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install -y cuda-toolkit-13-3

# Make sure nvcc is on PATH for this shell
export PATH=/usr/local/cuda/bin:$PATH
nvcc --version

Build llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=native

cmake --build build --config Release -j

Configure output you want to see:

-- Found CUDA: /usr/local/cuda (found version 13.3)
-- Cuda host compiler is /usr/bin/c++
-- Using CUDA architectures: 120

If the last line shows 120 (or your GPU’s capability), the build will produce kernels for it. Anything else means CMake did not detect your card — stop and fix that before compiling thousands of lines for the wrong target.

Why these options are needed

  • -DGGML_CUDA=ON enables the CUDA backend. Without it you get a CPU-only build no matter what GPU is plugged in.
  • -DCMAKE_CUDA_ARCHITECTURES=native — this is the Blackwell-specific bit. CMake runs nvcc against your local device and emits sm_120. On multi-GPU or headless build machines, replace native with an explicit list, e.g. -DCMAKE_CUDA_ARCHITECTURES="80;90;120" to also support older cards.
  • -j parallelizes the build. llama.cpp’s CUDA translation units are heavy; on a weak machine expect several minutes, most of it in nvcc.

Verify the build

./build/bin/llama-cli --version
./build/bin/llama-cli -m /path/to/model-q4_k_m.gguf \
  -ngl 99 \
  -p "Hello" \
  -n 16

Watch the startup log for a CUDA section similar to:

CUDA0: NVIDIA RTX PRO 5000 Blackwell (SM count: ... free: ... GB)
model loaded to GPU

If you see CUDA HGGC... fallback messages or no kernel image is available, jump to Common errors. A healthy run reports prompt processing (PP) and token generation (TG) speeds in the final lines. The only llama.cpp figures currently published for these cards are on the RTX PRO 5000 benchmark page — note they come from an in-progress PR build rather than the released toolchain described here, so expect your numbers to differ.

Common errors

“no kernel image is available for execution on the device”

Your binary has no SASS for this GPU. Almost always a missing/wrong CMAKE_CUDA_ARCHITECTURES. Rebuild with native, or confirm the configure log shows your architecture before compiling.

“Could NOT find CUDA” / “nvcc not found”

Either the toolkit isn’t installed, or CMake can’t see it. Fix the PATH (nvcc --version must work in the same shell that runs cmake) or pass -DCMAKE_CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda.

Build succeeds but runs on CPU only

Usually a -ngl problem, not a build problem. Pass -ngl 99 (all layers) and check the model-load log for “offloaded N/N layers to GPU”.

Slow nvcc, or OOM during compile

CUDA compilation is memory-hungry per translation unit. Lower parallelism (-j4) if you see the OOM killer, or enable ccache for rebuilds:

sudo apt install -y ccache
cmake -B build -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=native

Performance notes

  • Keep the whole model in VRAM (-ngl 99) and use a quantization that fits — see GPU Model Fit to estimate this before downloading.
  • KV cache type matters for long context: -ctk q8_0 -ctv q8_0 roughly halves KV memory with minimal quality loss.
  • Expect prompt processing to scale near-linearly with GPU count; token generation is memory-bandwidth bound on a single card, which is where Blackwell’s GDDR7 shows up most.

Sources and upstream documentation

Spotted a broken command or a wrong number?Email me — corrections get fixed on the page.