Downloading a model file smaller than your available memory is an encouraging start. It is not the end of the arithmetic. The runtime still needs space to do something with it, which is an unreasonable demand we have so far failed to negotiate away.
Separate model weights, key/value cache, working buffers, runtime overhead, and other allocations. Their locations also matter: system RAM and dedicated GPU memory are not interchangeable labels on one bucket.
Count a cache you can describe
For a simple full-attention decoder with an unquantized cache, a useful raw-storage estimate is:
KV bytes = 2 × layers × KV heads × head dimension × cached tokens × bytes per element × sequences.
The factor of two represents keys and values. This is a tensor-storage calculation, not a universal prediction of process memory. Cache implementation, allocation strategy, sliding-window behavior, offloading, and quantization can change the actual allocation. Hugging Face documents these cache strategies and their tradeoffs. Hugging Face
Use KV heads, not automatically the number of query attention heads. Derive head dimension from the model configuration only when that architecture permits the derivation.
Work one inspectable example
The retrieved Qwen2.5-0.5B-Instruct configuration records 24 layers, 14 attention heads, two KV heads, and hidden size 896. For this configuration, 896 ÷ 14 gives head dimension 64. Qwen
For an illustrative 4,096-token cache, one sequence, and two-byte elements:
2 × 24 × 2 × 64 × 4096 × 2 × 1
= 50,331,648 bytes
= 48 MiBAt 8,192 tokens the same raw estimate doubles to 96 MiB. That does not say the whole model uses 48 MiB or 96 MiB. It says one explicitly defined cache calculation produces those values.
Keep binary and decimal units visible. A GiB is 2³⁰ bytes; a GB is 10⁹ bytes. Do not subtract quantities displayed under different conventions and then blame quantization for the result.
Add only the terms you actually know
If you know the exact model artifact size, record its filename, format, quantization, and hash. A configuration file does not establish the byte size of every converted artifact. A simple parameters-times-bits estimate excludes format details and should be labeled as such.
Runtime working space should remain an explicit unknown until measured or established for the specific implementation. Do not bury it in a universal percentage and call the answer “required VRAM.” If a planning allowance is used, make it editable and name it as an allowance.
Offloading also changes where the memory lives and can affect performance. A model that starts is not necessarily one that responds at a useful speed. ggml-org
Close the loop with a measurement
Record the runtime version, backend, model hash, context configuration, cache types, and concurrent workload. Observe memory at startup and during a representative run. Preserve both host and accelerator readings when applicable, including how they were measured.
Compare the observation with the component estimate. The difference is a clue to investigate, not evidence that the arithmetic should be adjusted until it predicts the number you wanted.
A useful memory workbench explains the budget and its unknowns. It should never congratulate a visitor on compatibility merely because the most convenient term fits.
Next: Tokens per second need footnotes explains what to measure after the model starts.
Sources
- Cache strategies · Hugging Face — Hugging Face. Reviewed 15 September 2026.
- Qwen2.5-0.5B-Instruct model configuration — Qwen. Reviewed 15 September 2026.
- llama-server — ggml-org. Reviewed 15 September 2026.
Put this to work
- Memory Workbench — Calculate a full-attention decoder’s raw cache storage; keep unknown runtime allocations visible.