AI & Compute

Build llama.cpp with CUDA for NVIDIA Blackwell

Compile llama.cpp for Blackwell GPUs (sm_120) with a current CUDA toolkit — correct architecture flags, build commands, and fixes for the errors you will actually hit.

Updated August 21, 2026Tested with Ubuntu 26.04 · 13.3 · RTX PRO 5000 Blackwell · 595.x (CUDA 13.3 compatible) · llama.cpp b8f2c41 + CMake 3.31, Ninja

The prebuilt binaries of llama.cpp rarely include the right SASS for brand-new GPUs, and “just add CUDA” is not enough: Blackwell needs an explicit compute capability or the kernel image will simply not be compiled in. This guide gets a working, GPU-accelerated build on Ubuntu with a current toolkit.

Prerequisites

  • GPU: NVIDIA Blackwell with a datacenter, workstation or gaming SKU (RTX PRO 5000/6000, RTX 50xx series). Compute capability sm_120 for workstation/consumer; sm_100 for B100/B200/GB200 class.
  • Driver: recent enough for your CUDA toolkit version — check with nvidia-smi, which prints the supported CUDA release in the top-right corner.
  • CUDA toolkit (optional but recommended): a system install of the cuda-toolkit package, or let CMake find one you point at via CUDA_HOME.
  • Build tools: CMake 3.24+, Ninja (or make), a C++17 compiler, git.

Install the CUDA toolkit

llama.cpp needs nvcc from a toolkit version that knows about your architecture. On Ubuntu:

# Add the NVIDIA repository (skip if you already have CUDA installed)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install -y cuda-toolkit-13-3

# Make sure nvcc is on PATH for this shell
export PATH=/usr/local/cuda/bin:$PATH
nvcc --version

Build llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=native

cmake --build build --config Release -j

Configure output you want to see:

-- Found CUDA: /usr/local/cuda (found version 13.3)
-- Cuda host compiler is /usr/bin/c++
-- Using CUDA architectures: 120

If the last line shows 120 (or your GPU’s capability), the build will produce kernels for it. Anything else means CMake did not detect your card — stop and fix that before compiling thousands of lines for the wrong target.

Why these options are needed

  • -DGGML_CUDA=ON enables the CUDA backend. Without it you get a CPU-only build no matter what GPU is plugged in.
  • -DCMAKE_CUDA_ARCHITECTURES=native — this is the Blackwell-specific bit. CMake runs nvcc against your local device and emits sm_120. On multi-GPU or headless build machines, replace native with an explicit list, e.g. -DCMAKE_CUDA_ARCHITECTURES="80;90;120" to also support older cards.
  • -j parallelizes the build. llama.cpp’s CUDA translation units are heavy; on a weak machine expect several minutes, most of it in nvcc.

Verify the build

./build/bin/llama-cli --version
./build/bin/llama-cli -m /path/to/model-q4_k_m.gguf \
  -ngl 99 \
  -p "Hello" \
  -n 16

Watch the startup log for a CUDA section similar to:

CUDA0: NVIDIA RTX PRO 5000 Blackwell (SM count: ... free: ... GB)
model loaded to GPU

If you see CUDA HGGC... fallback messages or no kernel image is available, jump to Common errors. A healthy run reports prompt processing (PP) and token generation (TG) speeds in the final lines — for the exact numbers we measured on this hardware, see RTX PRO 5000 Blackwell benchmarks.

Common errors

“no kernel image is available for execution on the device”

Your binary has no SASS for this GPU. Almost always a missing/wrong CMAKE_CUDA_ARCHITECTURES. Rebuild with native, or confirm the configure log shows your architecture before compiling.

“Could NOT find CUDA” / “nvcc not found”

Either the toolkit isn’t installed, or CMake can’t see it. Fix the PATH (nvcc --version must work in the same shell that runs cmake) or pass -DCMAKE_CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda.

Build succeeds but runs on CPU only

Usually a -ngl problem, not a build problem. Pass -ngl 99 (all layers) and check the model-load log for “offloaded N/N layers to GPU”.

Slow nvcc, or OOM during compile

CUDA compilation is memory-hungry per translation unit. Lower parallelism (-j4) if you see the OOM killer, or enable ccache for rebuilds:

sudo apt install -y ccache
cmake -B build -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=native

Performance notes

  • Keep the whole model in VRAM (-ngl 99) and use a quantization that fits — see GPU Model Fit to estimate this before downloading.
  • KV cache type matters for long context: -ctk q8_0 -ctv q8_0 roughly halves KV memory with minimal quality loss.
  • Expect prompt processing to scale near-linearly with GPU count; token generation is memory-bandwidth bound on a single card, which is where Blackwell’s GDDR7 shows up most.

Sources and upstream documentation