“It does 80 tokens per second” is a beginning, not a benchmark. Which model? Which quantization? Which phase? Which context? Which machine? Were three other things occupying the GPU, or had the browser merely become unusually ambitious?
A useful benchmark result is a measurement plus the conditions that produced it. Without those conditions, the number is mostly a conversation starter.
Separate the phases
llama-bench distinguishes prompt processing, text generation, and combined tests. It supports machine-readable output including JSON and CSV, repeats tests, and reports summary statistics. Read the help for your installed build because options can evolve. ggml-org
Do not place a prompt-processing rate and a generation rate in the same ranking column. Do not treat either as a direct measure of answer quality. Throughput, latency, and task success answer different questions.
Make one reproducible run
For a local build whose help confirms these options, a starting command is:
llama-bench -m /path/to/model.gguf -p 512 -n 128 -r 5 -o jsonThis is a procedure, not a reported result. Keep the complete output. Record the executable’s version or commit, the model’s hash, the backend and device, thread counts, offload settings, cache types, and relevant batch settings. Capture the actual command rather than reconstructing it from memory later.
Run on an otherwise stable machine and note background load. If the objective is to compare configurations, choose the variable you intend to change and keep the rest matched. For a more complex experiment, consult an appropriate design rather than assuming one-variable changes reveal all interactions. NIST/SEMATECH
Compare compatible records
Group your exported records by an explicit comparison key. At minimum, inspect model identity, quantization, test phase, prompt and generation lengths, backend, and relevant runtime settings. Missing fields should produce “comparison incomplete,” not quietly match every other row with a blank value.
Compare different environments side by side, but list which fields differ before interpreting any throughput ratio. That makes it possible to learn from imperfect evidence without pretending it is cleaner than it is.
Keep individual repetitions when available. If five hypothetical rates are 20, 20, 21, 20, and 8, the slow run deserves investigation. Removing it because four other runs look nicer changes the experiment. Label any exclusions and retain the original rows.
Measure the application separately
A chat application includes work beyond a benchmark’s isolated kernels or phases. Model loading, prompt preparation, queueing, retrieval, tool calls, and output handling may affect the experience. Server behavior and its reported timing fields need to be interpreted according to the runtime documentation. ggml-org
For a user-facing task, record a task-level outcome as well: did the response arrive, did it satisfy the requested format, and did it pass a relevant correctness check? A faster wrong answer is occasionally convenient, but rarely the intended feature.
Keep the receipt
Keep the original measurement output alongside any normalized spreadsheet, parsing warnings and comparison criteria. Mark documentation examples as examples and your own measurements as measurements. Never relabel numbers from a README as results from the visitor’s computer.
The best benchmark chart lets another builder understand why two bars are comparable. If the chart cannot do that, it has successfully plotted numbers and has some work left before becoming evidence.
Next: The weights fit. The model doesn’t. separates memory accounting from throughput.
Sources
- llama-bench — ggml-org. Reviewed 15 September 2026.
- 5.3. Choosing an experimental design — NIST/SEMATECH. Reviewed 15 September 2026.
- llama-server — ggml-org. Reviewed 15 September 2026.
Put this to work
- Memory Workbench — Calculate a full-attention decoder’s raw cache storage; keep unknown runtime allocations visible.