llama.cpp Common Options

-ngl / —gpu-layers / —n-gpu-layers

Maximum number of transformer layers to store in VRAM. Takes an exact number, auto (default) or all. Conventional values like 99 or 999 ask for more layers than a model has, which is how they ended up meaning “everything”; saying all states that intent directly. Lower the number when the load runs out of memory — the load log prints how many layers actually landed on the GPU.

—tensor-split

Comma-separated proportions (relative weights, not percentages) that distribute tensors across the devices listed in --device, e.g. --tensor-split 3,2 gives the first device 60% of the share. The colon form (48:24) does not split — it parses as a single value and silently puts everything on the first device.

-ts syntax: llama-server vs llama-bench (this one bites)

In llama-bench, a comma selects multiple benchmark variants, and a slash (or semicolon) separates devices within one split. In llama-server / llama-cli both characters are accepted as device separators. So the same string means different things:

Checked against the source rather than folklore: tools/llama-bench/llama-bench.cpp splits the -ts value on , into variants (split_delim = ','), then each variant on [;/]+ into per-device weights, padding missing devices with 0.0; common/arg.cpp splits the same option on [,/]+. A -ts 1,1 sweep therefore silently produces two single-GPU rows instead of one split run — and a tensor-parallel study that looks like it did the experiment ran two baselines.

A capacity ratio is not automatically the fastest split: 72 + 48 GB reduces to 3/2, but an equal 1/1 tensor split beat intuition on exactly that pair of cards (+61% generation). Measure 1/1 whenever half the allocation fits on the smaller card; leave several GiB per GPU for compute buffers, KV cache and driver overhead either way.

GGML_CUDA_P2P

Environment variable. In current llama.cpp, direct CUDA peer access is opt-in at runtime — set it and the backend enables peer access between GPUs that report capability:

GGML_CUDA_P2P=1 llama-server -m model.gguf --device CUDA0,CUDA1 -sm tensor

The test in ggml/src/ggml-cuda/ggml-cuda.cu is getenv("GGML_CUDA_P2P") != nullptr, so the value is irrelevant — the variable being set is what enables it: GGML_CUDA_P2P=1, =yes and even =0 all enable it. Use =1 by convention, and turn peer access off by removing the variable — export GGML_CUDA_P2P=0 does not mean false. Note also that an NCCL-enabled build takes the peer-access path for VMM allocations regardless of this variable, so “I never set it” is not proof a run avoided peer DMA.

Upstream’s own caveat, worth taking seriously before blaming a model: peer access “may cause crashes or corrupted outputs on some motherboards or BIOS configurations (e.g. when IOMMU is enabled)” — corrupt token output can be a symptom of the transport, not only of the model. See Fix NVIDIA Multi-GPU P2P Corruption on Intel Linux.

GGML_CUDA_NO_PEER_COPY

CMake option (option(GGML_CUDA_NO_PEER_COPY … OFF) in ggml/CMakeLists.txt), not a runtime switch. It compiles out the direct cudaMemcpyPeerAsync paths, so inter-GPU data is staged through host memory instead:

cmake -B build-cuda -DGGML_CUDA=ON -DGGML_CUDA_NO_PEER_COPY=ON && cmake --build build-cuda -j

Value as a diagnostic control: if a normal build gives corrupt output across two GPUs and this build is stable on the same weights, quant, seed and prompt, the fault is in the peer DMA path rather than in the model. It also disables the backend’s copy/compute-overlap capability, so expect some slowdown for a genuine reason — do not benchmark it as if it were the same build. Not to be confused with GGML_CUDA_NO_PINNED.

GGML_CUDA_NO_PINNED

Environment variable that disables pinned host memory (the runtime check is again “set at all”, so =0 still turns it on). It is about host staging buffers, not GPU peer access: turning pinned memory off will not route you around a broken peer-copy path — if anything it makes host-staged transfers slower. Not the fix for P2P corruption.

GGML_CUDA_NCCL / NCCL_P2P_DISABLE

-DGGML_CUDA_NCCL=ON is a build-time option (default ON) with no runtime flag; when present, llama.cpp uses NCCL for cross-GPU reductions in --split-mode tensor. It is not distributed with CUDA — check the CMake output, and watch for this warning at startup:

NVIDIA Collective Communications Library (NCCL) is unavailable, multi GPU performance will be suboptimal

NCCL_P2P_DISABLE=1 changes how NCCL moves data between ranks. It does not disable llama.cpp’s own use of the CUDA peer-copy APIs, so it is not a way to test whether the peer path is at fault — that is what GGML_CUDA_P2P (runtime) and GGML_CUDA_NO_PEER_COPY (build-time) are for. ROCm’s equivalent is -DGGML_HIP_RCCL=ON, off by default.

—main-gpu

Id of the main GPU (receives prompt processing and the output embeddings). Use it when your cards are asymmetric, e.g. put it on the 72 GB card.

—split-mode

none, layer, row, or tensor (help text at the commit checked below). What the modes actually do is the part worth internalising, because the performance difference between them is usually communication cost:

• layer (default) — whole layers go to each card. Communication is limited to activations at layer boundaries; the cards pipeline, they do not compute the same layer together. This is why a slow inter-card link can be nearly invisible in layer mode.
• row — splits each layer’s weights by rows across cards (parallelized).
• tensor — splits weights and KV cache across cards; every card computes part of every layer and reduces across GPUs. Parallelized, EXPERIMENTAL, and the mode where peer bandwidth and P2P correctness actually matter.

Hard requirements for -sm tensor (upstream multi-GPU docs): flash attention must be on, and the KV cache types must be non-quantized — pass -ctk f16 -ctv f16 (or bf16/f32). Quantized KV plus tensor mode is a startup error (“simultaneous use of SPLIT_MODE_TENSOR and KV cache quantization not implemented”), not a slow path.

Support varies by architecture and revision. At llama.cpp 4a8993735 this is implemented as a blocklist in llm_arch_supports_sm_tensor() — most architectures are supported, and Grok, MPT, OLMoE, DeepSeek2/32, Nemotron-H(-MoE), Granite-Hybrid, Minimax-M2/M3, Mistral4, Kimi-Linear/K3, Jamba, Falcon-H1, Mamba/Mamba2, PLAMO2, MiniCPM3, Gemma-3n, OLMo2, BitNet, T5, Qwen3TTS and Qwen4Exp are excluded. An unsupported architecture fails at load with LLAMA_SPLIT_MODE_TENSOR not implemented for architecture '…'; the list moves between releases, so check your build.

Dated note (2026-09): two large models (Qwen3.8 Flash Next, Laguna S 2.1) crashed when split across a pair of RTX PRO 5000s on a machine that was logging IOMMU faults against GPU DMA writes at the time. On a faulting peer path a split-mode crash is a platform symptom, so those two cases are platform evidence — and re-measuring them on a healthy path is the next step before anything is concluded about splitting. Before blaming the split implementation for a two-GPU crash or garbled output, check dmesg for DMAR faults — see Fix NVIDIA Multi-GPU P2P Corruption on Intel Linux.

—override-tensor

Force tensors matching a name pattern onto a buffer type — syntax is pattern=buffer, comma-separated for several: --override-tensor "output=CPU". Escape hatch when auto-placement makes bad choices.

—cpu-moe

Keep MoE expert layers on the CPU and run only attention on GPU. Useful for MoE models whose experts don’t fit in VRAM.

-c / —ctx-size

Maximum context length in tokens. KV cache is allocated for this size up front — the main driver of long-context OOMs.

-ctk

Key-cache quantization type (e.g. f16, q8_0, q4_0). q8_0 roughly halves key memory with minimal quality impact.

-ctv

Value-cache quantization type — same options and trade-offs as -ctk. Set both to keep the pair consistent.

-fa / —flash-attn

Flash attention: on, off, or auto ([on|off|auto] in --help). Reduces KV memory pressure and speeds long contexts on capable GPUs. -sm tensor requires it: with auto llama.cpp enables it and logs enabling flash_attn since it is required for SPLIT_MODE_TENSOR; with explicit -fa off the run stops with SPLIT_MODE_TENSOR requires flash_attn to be enabled.

—no-kv-offload

Keep the KV cache on CPU instead of GPU. A last-resort OOM fix — token generation slows down significantly.

-t / —threads

CPU thread count for the CPU portion of the work. Set to physical core count; hyperthreading above that rarely helps and can hurt.

-b / —batch-size

Logical maximum prompt-processing batch size. Larger batches improve PP throughput up to the point where memory or the backend saturates.

-ub / —ubatch-size

Micro-batch size within -b. Tune downward if prompt processing OOMs at high -b.

-lm / —load-mode

How the model file is read into memory: auto, none, mmap, mlock, mmap+mlock or dio. mlock keeps pages resident so they cannot be swapped; mmap+mlock maps the file and locks it; none reads without mapping. At the checked commit this replaces the older --no-mmap and -mlock flags, which are no longer registered in common/arg.cpp — on builds that still have them, --no-mmap -mlock corresponds to --load-mode mlock.

—prio

Process/thread priority on the CPU side: -1 low, 0 normal, 1 medium, 2 high, 3 realtime. Values outside that range are rejected. Use it when inference competes with other heavy local work.

—temp / —temperature

Sampling temperature. Lower (0.1–0.3) makes output deterministic; higher adds diversity. For fact-heavy or tool-calling models, keep it low.

—top-k

Keep only the top k tokens per step. 40 is a common starting point; 1 is greedy decoding.

—top-p

Nucleus sampling threshold (e.g. 0.9). Applied after top-k; don’t stack aggressive values of both without checking results.

—min-p

Remove tokens whose probability is below a fraction of the top token’s (e.g. 0.05). Often the smoothest single knob for general chat.

—repeat-penalty

Penalty against repeating recently generated text (e.g. 1.1). Raise if output loops; lower if it starts saying too little.

-s / —seed

Fix the RNG seed for reproducible runs. Combined with low temperature, identical inputs give identical outputs.

-v / —verbose

Increase log detail. Useful while tuning offload/VRAM: it prints device, layer placement, and KV cache allocation.

—log-disable

Turn off the per-token progress spam during long generations without losing error messages.

—version

Print build info including commit — record this next to any benchmark you publish so others can reproduce your setup.