Results gallery
Real output from real machines. Each directory holds exactly what llama-roofline run
writes: roofline.json (machine-readable), report.txt (the terminal report card),
report.md (shareable), and roofline.png.
Two devices at opposite ends of the range are included here. If you run the tool, please add yours: open an issue with the "Results gallery" template. Hardware I do not own is the most useful thing anyone can contribute, and results that disagree with the model are more interesting than results that confirm it.
| device | memory | models | ceiling | fitted BW_eff | R² | utilisation | verdict |
|---|---|---|---|---|---|---|---|
| Intel i7-12700H | DDR5-4800, 34 GB | 7 (0.5B to 7B) | 53.9 GB/s | 37.65 GB/s | 0.987 | 65-97% | memory-bound |
| Raspberry Pi 5 (Cortex-A76) | LPDDR4X, 2 GB | 7 (0.5B to 1.5B) | 13.98 GB/s | 10.69 GB/s | 0.980 | 73-86% | memory-bound |
The two machines differ by 3.5x in fitted bandwidth and by roughly 20x in price, and both
land in the same place: decode sits between 65% and 97% of the memory ceiling, and
throughput tracks 1/model_bytes with an R² above 0.98. That is the point of the tool.
The roofline is not a property of any one device.
Two details worth reading off these:
- On the laptop the largest model runs closest to the wall (the 7B at 97% of ceiling, the 0.5B Q4 at 65%). Bigger working sets stream more efficiently and amortise the per-token overhead that is not weight traffic.
- Using every hardware thread is a trap on both machines. On the laptop,
-t 20cost up to 63% of decode throughput versus-t 8. On the Pi, using all 4 cores cost up to 21% versus 2. Prefill, meanwhile, kept scaling in both cases.
About the Raspberry Pi 5 numbers
These were not produced by running llama-roofline on the Pi. They are the raw
llama-bench measurements from the study this tool grew out of, re-expressed in the
tool's schema and re-analysed by the tool's own code, using
rpi5-cortex-a76/convert_paper_data.py. Read the
docstring at the top of that script for the full provenance note; the differences from a
native run (file size instead of model_size for bytes-per-token, an externally supplied
bandwidth ceiling) are recorded in the results JSON and printed in the report's caveats.
The conversion is also a correctness check: handed the study's raw data, the tool recovers the published fit of 10.69 GB/s at R² = 0.980 over 7 points, exactly.
Reproducing these
# x86 example
llama-roofline run --models /path/to/models --threads 1,4,8,14,20 --reps 3
# Pi 5 example, from the published data
cd examples/rpi5-cortex-a76
python convert_paper_data.py /path/to/edge-llm/results
rpi5-cortex-a76
Terminal report
========================================================================
llama-roofline v0.1.0 -- report card
========================================================================
Machine : Raspberry Pi 5 (Broadcom BCM2712, Cortex-A76 @ 2.4 GHz)
4 logical / 4 physical cores, 2.0 GB RAM, Linux aarch64
Memory : 14.0 GB/s (external STREAM-style read benchmark for this board)
llama.cpp : build ?, backends: CPU
IS YOUR DECODE MEMORY-BOUND?
------------------------------------------------------------------------
YES -- your decode is memory-bandwidth-bound.
Token generation is running at 77% of the memory bandwidth this
machine can actually sustain. The CPU spends most of each token
waiting for weights to arrive from RAM, not computing.
THE ROOFLINE
------------------------------------------------------------------------
decode tok/s = 10.69 GB/s / model bytes
fitted across 7 models, R^2 = 0.9800,
taking each model at its own best thread count
that effective bandwidth is 76% of your 14.0 GB/s ceiling
A fit this tight means one number -- bytes -- predicts your
generation speed. You can size a model for a target tok/s.
WHAT TO DO ABOUT IT
------------------------------------------------------------------------
* Model size is your throughput dial. Halving the bytes you load
roughly doubles tok/s: a smaller model or a lower quant buys speed
almost exactly in proportion to the bytes it removes.
* Measured on your models: Qwen2.5-0.5B Q2_K is 2.91x smaller than
Qwen2.5-1.5B Q4_K_M and decodes 2.55x faster. Bytes in, tokens out.
* Best decode thread count: 2. Past that, extra threads add
contention, not throughput: decode is waiting on memory, and more
waiters do not make the memory faster.
* Using every thread cost up to 21% of decode throughput versus the
best setting. Set -t explicitly; do not let it default.
* Prefill is different: it scaled 3.3x with threads (median across
your models). Prompt processing is compute-bound, so long-prompt
workloads DO want all your cores even though generation does not.
YOUR MODELS
------------------------------------------------------------------------
model quant size decode prefill thr GB/s %ceil
------------------------------------------------------------------------------------
Qwen2.5-0.5B Q2_K Q2_K 339 MB 31.1t 148t 4 10.5 75%
Qwen2.5-0.5B Q3_K_M Q3_K_M 355 MB 29.1t 197t 4 10.4 74%
Qwen2.5-0.5B Q4_K_M Q4_K_M 398 MB 28.1t 89t 3 11.2 80%
Qwen2.5-0.5B Q5_K_M Q5_K_M 420 MB 25.5t 79t 4 10.7 77%
Qwen2.5-0.5B Q8_0 Q8_0 531 MB 19.1t 249t 4 10.1 73%
Llama-3.2-1B Q4_K_M Q4_K_M 808 MB 14.4t 62t 2 11.6 83%
Qwen2.5-1.5B Q4_K_M Q4_K_M 986 MB 12.2t 55t 2 12.0 86%
CAVEATS
------------------------------------------------------------------------
! converted data: measurements come from the published Raspberry Pi 5
study, not from a native llama-roofline run. Bytes-per-token uses
the GGUF file size rather than llama-bench's model_size, so
utilisation reads ~1-2% high. See convert_paper_data.py for the full
provenance note.
* bytes-per-token is taken as the model's resident size. That is exact
for a dense transformer at short context and an overestimate once
the KV cache grows large, or for MoE models.
* The ceiling was supplied rather than measured by this tool, so every
percentage here is only as good as that number.
* Numbers are throughput only. Nothing here measures output quality.
========================================================================
Files: convert_paper_data.py, report.md, report.txt, roofline.json, roofline.png
x86-i7-12700H
Terminal report
========================================================================
llama-roofline v0.1.0 -- report card
========================================================================
Machine : 12th Gen Intel(R) Core(TM) i7-12700H
20 logical / 14 physical cores, 34.0 GB RAM, Windows AMD64
Memory : 53.9 GB/s sustained read (measured: dot kernel, 10 threads, 512 MB set)
llama.cpp : build 10154 (0e4a03622), backends: CPU
IS YOUR DECODE MEMORY-BOUND?
------------------------------------------------------------------------
YES -- your decode is memory-bandwidth-bound.
Token generation is running at 72% of the memory bandwidth this
machine can actually sustain. The CPU spends most of each token
waiting for weights to arrive from RAM, not computing.
THE ROOFLINE
------------------------------------------------------------------------
decode tok/s = 37.65 GB/s / model bytes
fitted across 7 models, R^2 = 0.9874,
taking each model at its own best thread count
that effective bandwidth is 70% of your 53.9 GB/s ceiling
A fit this tight means one number -- bytes -- predicts your
generation speed. You can size a model for a target tok/s.
WHAT TO DO ABOUT IT
------------------------------------------------------------------------
* Model size is your throughput dial. Halving the bytes you load
roughly doubles tok/s: a smaller model or a lower quant buys speed
almost exactly in proportion to the bytes it removes.
* Measured on your models: qwen0.5b-q2k is 14.06x smaller than
qwen7b-q4km and decodes 10.15x faster. Bytes in, tokens out.
* Best decode thread count: 8. Past that, extra threads add
contention, not throughput: decode is waiting on memory, and more
waiters do not make the memory faster.
* Using every thread cost up to 63% of decode throughput versus the
best setting. Set -t explicitly; do not let it default.
* Prefill is different: it scaled 6.7x with threads (median across
your models). Prompt processing is compute-bound, so long-prompt
workloads DO want all your cores even though generation does not.
YOUR MODELS
------------------------------------------------------------------------
model quant size decode prefill thr GB/s %ceil
------------------------------------------------------------------------------------
qwen0.5b-q2k Q2_K 333 MB 113.3t 517t 8 37.7 70%
qwen0.5b-q4km Q4_K_M 392 MB 89.3t 351t 8 35.0 65%
qwen0.5b-q8 Q8_0 525 MB 78.6t 294t 14 41.3 76%
llama1b-q4km Q4_K_M 800 MB 48.2t 245t 14 38.6 72%
qwen1.5b-q4km Q4_K_M 980 MB 39.3t 179t 14 38.5 71%
qwen3b-q4km Q4_K_M 1.92 GB 21.0t 87t 8 40.3 75%
qwen7b-q4km Q4_K_M 4.68 GB 11.2t 42t 20 52.2 97%
CAVEATS
------------------------------------------------------------------------
* bytes-per-token is taken as the model's resident size. That is exact
for a dense transformer at short context and an overestimate once
the KV cache grows large, or for MoE models.
* The ceiling is what this machine sustained under a portable
microbenchmark, not the spec sheet number. It is a floor on the
truth, so the percentages are conservative.
* Numbers are throughput only. Nothing here measures output quality.
========================================================================
Files: report.md, report.txt, roofline.json, roofline.png
