Choose a model and quantization, then enter the context length and available memory. The estimate includes model weights, KV cache, and an allowance for runtime overhead.
How to read the estimate
Model weights: For presets with a recorded GGUF size, the calculator uses that file size. Other models use an estimate based on parameter count and quantization.
KV cache: Memory depends on context length, cache type, and model architecture. The result indicates whether the calculator has enough model information to calculate it or is using an estimate.
Runtime overhead: The calculator includes an allowance for working memory. Actual use varies with the software and configuration.
Treat a result close to your memory limit as a reason to check the model’s actual loading behavior. A model file fitting in memory does not mean the full workload will fit.
MoE models
Mixture-of-experts models report total parameters (all experts), while inference speed scales with active parameters per token. Memory must still hold every expert, so the fit estimate always uses total parameter weight size. The tool shows both numbers when they are known.
Memory observed on my Strix Halo system
For models I have actually run on my test machine, the tool shows a separate
Measured on How Quant test hardware card: amdgpu.gtt usage for the
inference process on a Corsair AI Workstation 300 (Ryzen AI Max+ 395, 128 GiB
LPDDR5), per backend (ROCm and Vulkan RADV), with the test date.
These readings show GPU-visible allocations reported by amdgpu. They are not
the process’s total memory use — a process can hold more in CPU-side state and
framework overhead. Some earlier runs did not record the exact generation
length; those cards carry a context label on the card itself.
What this tool does not do
- Predict speed. Tokens/s depends on compute path, quantization details, and memory bandwidth in ways a size check cannot capture. For speed, see the Strix Halo ROCm vs Vulkan benchmark and the RTX PRO 5000 benchmark.
- Cover every quantization for every model. Custom quant labels (UD/IQ variants) work as inputs; their effective bits/weight come from a small reference table with a conservative fallback, labeled Estimated in the notes.
- Model multi-GPU or partial offload strategies. The result is a single-device estimate; exceeding one GPU’s memory does not mean an offload setup will work on its own.
Calculation details — formulas and assumptions
Weights. For preset models the calculator uses the exact byte size of the GGUF artifact from the Hugging Face repository, pinned at catalog-refresh time together with the repository revision it was observed at. Parameter × bits-per-weight estimates are only used when you enter a custom model without an exact size, and then they are labeled Estimated (with the effective bits/weight shown in the notes).
KV cache. If the model’s architecture metadata is available (layers, KV
heads, head dimension — captured at build time from the upstream
config.json), KV memory is Calculated exactly:
2 × layers × kv_heads × head_dim × context × bytes-per-element, where the
bytes per element follow your K/V cache type selection (F16/BF16 = 2 B,
Q8_0 = 1 B + scale, Q4_0 = 0.5 B + scale). Without architecture metadata (some
presets and all custom models) the tool falls back to a wide range scaled from
total parameter count, labeled Estimated.
Runtime overhead. A conservative fixed allowance (2–5 GiB) covers compute buffers and process memory overhead. It is deliberately a range: the real figure depends on backend, allocator behavior, and build flags.
Result bands. Headroom is judged against available memory with an uncertainty band: ≥20% remaining → Comfortable fit; 10–20% → Likely fits; 0–10% → Tight fit; a small overshoot (≤10%) → may need offload or shorter context; a larger overshoot → Exceeds available memory.
Real deployments vary: allocator fragmentation, the serving framework’s bookkeeping, prompt tokenization edge cases, and extra features (tools, images) all shift usage by hundreds of MiB or more. The calculator answers plausibility, not certainty — which is what any pre-download check can provide.