What this page answers: what the llama.cpp flags you will actually use do — grouped by job, one line each.
Flags are grouped by job. Use the filter above to find one quickly; group
headings hide themselves when nothing in them matches. Note that flag names
shift between releases — llama-cli --help on your build is always the source
of truth.
GPU & Offloading
-ngl / —n-gpu-layers
Number of transformer layers to offload to the GPU. 99 offloads everything the backend can fit. Lower it when you hit OOM during load.
—tensor-split
Comma-separated proportions (relative weights, not percentages) that distribute tensors across the devices listed in --device, e.g. --tensor-split 3,2 gives the first device 60% of the share. The colon form (48:24) does not split — it parses as a single value and silently puts everything on the first device.
—main-gpu
Id of the main GPU (receives prompt processing and the output embeddings). Use it when your cards are asymmetric, e.g. put it on the 72 GB card.
—split-mode
none, layer, row, or tensor. Layer mode assigns whole layers to cards; row mode splits each layer’s matrices across cards; tensor mode places individual tensors (finest granularity). The RTX PRO 5000 benchmark used --split-mode tensor --tensor-split 3,2 on asymmetric 72 + 48 GB cards.
—override-tensor
Force tensors matching a name pattern onto a buffer type — syntax is pattern=buffer, comma-separated for several: --override-tensor "output=CPU". Escape hatch when auto-placement makes bad choices.
—cpu-moe
Keep MoE expert layers on the CPU and run only attention on GPU. Useful for MoE models whose experts don’t fit in VRAM.
Memory & Context
-c / —ctx-size
Maximum context length in tokens. KV cache is allocated for this size up front — the main driver of long-context OOMs.
-ctk
Key-cache quantization type (e.g. f16, q8_0, q4_0). q8_0 roughly halves key memory with minimal quality impact.
-ctv
Value-cache quantization type — same options and trade-offs as -ctk. Set both to keep the pair consistent.
-fa / —flash-attn
Flash attention: on, off, or auto. Reduces KV memory pressure and speeds long contexts on capable GPUs; leave on auto unless debugging.
—no-kv-offload
Keep the KV cache on CPU instead of GPU. A last-resort OOM fix — token generation slows down significantly.
Performance
-t / —threads
CPU thread count for the CPU portion of the work. Set to physical core count; hyperthreading above that rarely helps and can hurt.
-b / —batch
Maximum prompt processing batch size. Larger batches improve PP throughput up to the point where memory or the backend saturates.
-ub / —ubatch
Micro-batch size within -b. Tune downward if prompt processing OOMs at high -b.
—no-mmap
Load the model file fully into RAM instead of memory-mapping it. Faster and more stable for frequently re-accessed weights; uses real RAM.
-mlock
Lock the model into RAM so pages are never swapped out. Deprecated in current builds in favor of --load-mode, which covers the same cases: --load-mode mlock (mmap + lock) or --load-mode mmap+mlock; on constrained boxes this beats the older --no-mmap -mlock workaround.
—prio
Process priority for the CPU side (0 normal, 1 medium, 2 high). Use when inference competes with other heavy local work.
Sampling & Generation
-temp
Sampling temperature. Lower (0.1–0.3) makes output deterministic; higher adds diversity. For fact-heavy or tool-calling models, keep it low.
-top-k
Keep only the top k tokens per step. 40 is a common starting point; 1 is greedy decoding.
-top-p
Nucleus sampling threshold (e.g. 0.9). Applied after top-k; don’t stack aggressive values of both without checking results.
-min-p
Remove tokens whose probability is below a fraction of the top token’s (e.g. 0.05). Often the smoothest single knob for general chat.
-repeat-penalty
Penalty against repeating recently generated text (e.g. 1.1). Raise if output loops; lower if it starts saying too little.
—seed
Fix the RNG seed for reproducible runs. Combined with low temperature, identical inputs give identical outputs.
Misc
-v / —verbose
Increase log detail. Useful while tuning offload/VRAM: it prints device, layer placement, and KV cache allocation.
—log-disable
Turn off the per-token progress spam during long generations without losing error messages.
—version
Print build info including commit — record this next to any benchmark you publish so others can reproduce your setup.
Sources and upstream documentation
- llama.cpp repository — current flag definitions;
llama-cli --helpon your build is authoritative. - llama.cpp server documentation — server-side flags, including speculative decoding.