Example configs¶
Every sweep is one YAML file: the machines (devices), the model files (models),
and a matrix that expands into concrete runs. Preview any of them without running
anything:
mlsys run <config> --dry-run
Templates are written to be copied. Each has # <-- EDIT markers on the lines
that name your paths, and a header that says what to run before and after.
Studies are the exact configs behind the shipped datasets and write-ups, kept so a result can be traced to the sweep that produced it. Their paths point at the author's machines; use them as worked examples, or edit the device and model blocks to repeat a study on your own hardware.
The counts below come from expanding each file with the same loader mlsys run uses.
Templates¶
example-smoke.yaml¶
Example: minimal smoke test for a single local machine.
| file | configs/example-smoke.yaml |
| experiment | smoke-test |
| devices | local (local) |
| backend / mode | llamacpp / throughput |
| runs | 2 |
mlsys run configs/example-smoke.yaml --dry-run
onnx-cpu.yaml¶
ONNX Runtime on the CPU: latency percentiles and batch scaling for one model.
The onnxruntime backend times inference in-process on the device and reports mean, p50, p95 and p99 latency plus throughput, at each batch size. It needs onnxruntime in the interpreter the device points at (pip install ml-systems-lab[onnx] covers the local one); python: lets a device use a different environment from the harness.
| file | configs/examples/onnx-cpu.yaml |
| experiment | onnx-cpu |
| devices | local (local) |
| backend / mode | onnxruntime / throughput |
| runs | 12 |
mlsys run configs/examples/onnx-cpu.yaml --dry-run
quant-ladder.yaml¶
Which quantization of one model decodes fastest on this machine?
One local machine, one model in several GGUF quantizations, two thread counts. The question it answers is the one most people benchmark for: given the bytes each format streams per token, which one is fastest here, and how close does each get to the machine's bandwidth ceiling.
| file | configs/examples/quant-ladder.yaml |
| experiment | quant-ladder |
| devices | local (local) |
| backend / mode | llamacpp / throughput |
| runs | 8 |
mlsys run configs/examples/quant-ladder.yaml --dry-run
sbc-over-ssh.yaml¶
Benchmark a single-board computer from your laptop, over SSH.
The agent is copied to the board as source and runs there under the board's own Python (3.9 or newer, standard library only), so nothing is installed on the device and no network round trip sits inside a measurement. On a Raspberry Pi 5 the record also carries per-rail PMIC power, temperature and the throttle bits.
| file | configs/examples/sbc-over-ssh.yaml |
| experiment | sbc-over-ssh |
| devices | sbc (ssh) |
| backend / mode | llamacpp / throughput |
| runs | 4 |
mlsys run configs/examples/sbc-over-ssh.yaml --dry-run
thread-scaling.yaml¶
Where does decode stop scaling with threads on this CPU?
Decode is memory-bound, so throughput climbs with threads until the memory system is saturated and then flattens. On hybrid CPUs (performance plus efficiency cores) it can keep rising past the performance-core count, and past the peak the run-to-run spread widens sharply. Two model sizes and a fine thread sweep show where the peak sits.
| file | configs/examples/thread-scaling.yaml |
| experiment | thread-scaling |
| devices | local (local) |
| backend / mode | llamacpp / throughput |
| runs | 14 |
mlsys run configs/examples/thread-scaling.yaml --dry-run
ttft-latency.yaml¶
How long does a user wait for the first token, and how does that grow with the prompt?
Latency mode drives llama-server over its streaming API and times the first chunk (TTFT) and the last one (end-to-end), which is what an interactive user experiences. llama-bench measures neither. TTFT is mostly prefill, so it grows close to linearly with prompt length; this sweep shows the slope on your hardware.
| file | configs/examples/ttft-latency.yaml |
| experiment | ttft-latency |
| devices | local (local) |
| backend / mode | llamacpp / latency |
| runs | 3 |
mlsys run configs/examples/ttft-latency.yaml --dry-run
two-machines.yaml¶
One sweep across two machines at once: a laptop and a board over SSH.
With --concurrent the scheduler runs each device's queue in parallel, subject to each device's max_concurrency, so the sweep takes about as long as the slower machine rather than the sum of both. A clock check runs first and its offset is recorded. Only transport failures (a dropped SSH connection) are retried; a model that does not fit is a result, not a retry.
| file | configs/examples/two-machines.yaml |
| experiment | two-machines |
| devices | laptop (local), sbc (ssh) |
| backend / mode | llamacpp / throughput |
| runs | 6 |
mlsys run configs/examples/two-machines.yaml --dry-run
Studies¶
cloud-gpu.yaml¶
One cloud GPU node, joined to the home lab cluster, running the same sweep.
NOT YET EXECUTED. See k8s/cloud/RUNBOOK.md, which explains what has to happen first and in what order. The billing alert goes up before the instance, and the instance is destroyed in the session that created it.
| file | configs/cloud-gpu.yaml |
| experiment | cloud-gpu |
| devices | cloud-gpu (k8s), laptop-gpu (local) |
| backend / mode | llamacpp / latency, throughput |
| runs | 4 |
mlsys run configs/cloud-gpu.yaml --dry-run
cluster.yaml¶
Three genuinely different machines, one sweep, run at once.
This is the config that demonstrates the scheduler, as opposed to the containerisation study in configs/containerization.yaml, which deliberately arranges for the arms not to overlap because they share a CPU. Here they do not share anything, so they overlap, and the sweep takes about as long as its slowest device rather than the sum of all of them.
| file | configs/cluster.yaml |
| experiment | cluster |
| devices | laptop (local), pi5 (ssh), k8s-cpu4 (k8s) |
| backend / mode | llamacpp / throughput |
| runs | 6 |
| shipped records | results/cluster/ |
mlsys run configs/cluster.yaml --dry-run
cnn-smoke.yaml¶
Wiring check for the ONNX Runtime backend, mirroring configs/smoke.yaml for llama.cpp. Two architectures at two precisions is enough to catch a wrong interpreter, a missing provider or a model whose symbolic input shape did not resolve.
| file | configs/cnn-smoke.yaml |
| experiment | cnn-smoke |
| devices | laptop (local), pi5 (ssh), laptop-gpu (local) |
| backend / mode | onnxruntime / throughput |
| runs | 6 |
mlsys run configs/cnn-smoke.yaml --dry-run
containerization.yaml¶
What does containerisation cost an inference benchmark, and does it change the ranking?
Four arms, one machine, one binary.
| file | configs/containerization.yaml |
| experiment | containerization-tax |
| devices | wsl-host (ssh), k8s-unlimited (k8s), k8s-cpu8 (k8s), k8s-cpu4 (k8s), k8s-cpu2 (k8s) |
| backend / mode | llamacpp / throughput |
| runs | 25 |
| shipped records | results/containerization/ |
mlsys run configs/containerization.yaml --dry-run
int8-anomaly.yaml¶
Falsification study for the cross-ISA int8 observation.
Observed in the August 2026 campaigns on CIFAR-sized (32x32) inputs: int8 ResNet18 ran faster than fp32 on the Cortex-A76 and slower than fp32 on the i7-12700H, in the same runtime family. Treat that as an observation to be falsified, not as a result.
| file | configs/int8-anomaly.yaml |
| experiment | int8-anomaly |
| devices | laptop (local), pi5 (ssh) |
| backend / mode | onnxruntime / throughput |
| runs | 32 |
mlsys run configs/int8-anomaly.yaml --dry-run
int8-imagenet.yaml¶
H3: Does the signedness asymmetry hold at ImageNet scale (224x224)? ResNet50 v1 is 49x more FLOPS than CIFAR ResNet18 (32x32). If the effect disappears at scale, the CIFAR result is overhead-bound noise.
| file | configs/int8-imagenet.yaml |
| experiment | int8-imagenet |
| devices | laptop (local) |
| backend / mode | onnxruntime / throughput |
| runs | 8 |
mlsys run configs/int8-imagenet.yaml --dry-run
laptop-overnight.yaml¶
The laptop half of the matrix. RUN THIS WITH THE MACHINE IDLE: the same hardware measured 50.7 GB/s idle and 14.1 GB/s under background load, and no repetition count rescues a benchmark from constant contention.
| file | configs/laptop-overnight.yaml |
| experiment | laptop-overnight |
| devices | laptop (local) |
| backend / mode | llamacpp, onnxruntime / latency, throughput |
| runs | 75 |
| shipped records | results/laptop-campaign/ |
mlsys run configs/laptop-overnight.yaml --dry-run
pi-overnight.yaml¶
The full Pi 5 matrix. Spec coverage: 3 model sizes, 5 quantisations, thread sweep, prompt/output length sweeps, context-depth sweep, TTFT latency mode, ONNX Runtime as the second backend, and power measured in separate runs per the two-run rule.
Blocks are ordered smallest-model-first so an early failure costs the least, and every llama.cpp point resets swap and drops caches: on a 2 GB board, resident pages from the previous model turn the next run into an SD-card streaming benchmark.
| file | configs/pi-overnight.yaml |
| experiment | pi-overnight |
| devices | pi5 (ssh) |
| backend / mode | llamacpp, onnxruntime / latency, throughput |
| runs | 43 |
| shipped records | results/pi5-campaign/ |
mlsys run configs/pi-overnight.yaml --dry-run
smoke.yaml¶
A deliberately tiny matrix, used to check that a machine is wired up correctly before committing it to an overnight campaign. Two thread counts on the smallest model is enough to catch a wrong path, a missing binary or a dead SSH key, and it finishes in about a minute on the laptop.
| file | configs/smoke.yaml |
| experiment | smoke |
| devices | laptop (local), pi5 (ssh) |
| backend / mode | llamacpp / latency, throughput |
| runs | 6 |
mlsys run configs/smoke.yaml --dry-run
thread-cliff.yaml¶
Hypothesis: On Alder Lake hybrid CPUs (6P + 8E cores), LLM decode throughput peaks at the P-core count and collapses beyond it, with the magnitude of collapse inversely proportional to model size.
The i7-12700H has 6 P-cores (3.5-4.7 GHz, large L2) and 8 E-cores (2.3-3.5 GHz, shared L2). At t<=6, only P-cores are scheduled. At t>6, E-cores mix in. Decode is bandwidth-bound (matrix-vector); prefill is compute-bound (matrix-matrix). E-cores have lower IPC and smaller caches, which should hurt decode more than prefill.
| file | configs/thread-cliff.yaml |
| experiment | thread-cliff |
| devices | laptop (local) |
| backend / mode | llamacpp / throughput |
| runs | 33 |
mlsys run configs/thread-cliff.yaml --dry-run
Adding one¶
A config that answers a question other people ask is welcome as a pull request.
Put a template in configs/examples/, open it with a header comment whose first
paragraph says what question it answers, mark every machine-specific line with
# <-- EDIT, and run python tools/gen_config_gallery.py to add it here.