//Projects
Code, benchmarks and hardware.
A benchmarking platform, the reproducible artifact behind every paper, contributions upstream, and the hardware it all started on.
- Repositories
- 17
- Archived DOIs
- 15
- Merged upstream
- 8
- Open upstream
- 15
Platform
ML Systems Lab
One config file, one command, every machine on the bench.
A reproducible benchmarking framework for ML inference across heterogeneous hardware. One YAML config drives llama.cpp and ONNX Runtime sweeps on laptops, Raspberry Pi over SSH, GPUs, and Kubernetes pods, sequentially or across every machine at once; every run produces a self-describing record carrying the hardware, backend versions, measured metrics, the cgroup ceilings that applied, and the physical state of the machine while they were measured: power, temperature, clocks, throttle flags, utilization. The analysis layer turns a directory of records into publication-quality tables and figures.
Architecture
- 01 config.yaml machines, models, and the sweep matrix, declared once
- 02 RunSpecs deterministic run ids; interrupted campaigns resume by skipping what is on disk
- 03 Device local subprocess, agent pushed over SSH, or agent streamed into a Kubernetes pod; a new machine is a config block, not code, and no backend knows which kind it is talking to
- 04 Agent pure standard library, runs on the device under test; nothing crosses the network inside a measurement window
- 05 Scheduler optional concurrent sweeps with per-device limits, resource groups for devices that are secretly the same machine, circuit breakers, and retries only for transport failures
- 06 RunRecord raw output parsed on the host, so a parser bug is a re-parse, not a re-run; failures are records too, and a run under a CPU quota carries the quota it actually ran under
- 07 Observability Prometheus exporter and a checked-in Grafana dashboard for watching a sweep; the records stay the result, the exporter is never in the write path
- 08 Analysis text, Markdown and LaTeX tables; roofline and scaling figures
Supported hardware
- › Raspberry Pi 5, Cortex-A76: SSH agent, PMIC per-rail power, throttle bits, DVFS control
- › i7-12700H laptop, Windows: local agent, 20-thread sweeps, models up to 7B
- › RTX 3050 via ONNX Runtime DirectML: modeled as its own device
- › Kubernetes pods on k3s: cgroup ceilings read from inside the container, not copied from the config
Representative result
Reproduces the roofline fits of the decode-roofline paper exactly (Pi 5: 10.7 GB/s effective, R² = 0.980; i7-12700H: 35.7 GB/s, R² = 0.980), and two campaigns run natively by the framework re-measured the same quantities independently and agreed within 1.7%. Same config shows the GPU losing at batch 1 (dispatch overhead) and winning 29x at batch 64. Used it to measure what containerisation costs a benchmark: across 50 runs a pod is within +2.3% of a bare process and 14 of 15 comparisons cannot be separated from noise, while a CPU quota set below the thread count costs 64.1% and is resolved on every point. The ranking of five quantizations is identical in all five arms.
Related publications
- The Memory Wall at the Edge of Language
- The Memory Wall Governs Edge DNN Inference
- The Cold-Start Tax
- Kubernetes did not slow my benchmark down
Research artifacts
The harness, data and analysis behind each paper. Archived on Zenodo so a result can be cited and re-run.
edge-decode-thread-scaling
↗LLM decode thread scaling on a hybrid-core i7-12700H and a Raspberry Pi 5: the raw records, the analysis, and the macros the manuscript compiles from.
rpi5-quantization-benchmark
↗INT8 latency and energy measurements for nine networks on a Raspberry Pi 5 (Cortex-A76), released as raw records.
DOI 10.5281/zenodo.21844863
edge-llm-memory-wall
↗Decode-side roofline for on-device language models, plus the KV-cache capacity wall and an energy-aware serving policy.
Decode roofline R^2 = 0.98, on Arm and on x86
DOI 10.5281/zenodo.21844855
edge-cold-start-tax
↗Wake-transient characterisation for duty-cycled edge inference, including GD-Tax, a tax-aware cache policy that beats LRU and LFU.
5x to 23x cold-start tax, >80x past the eviction cliff
DOI 10.5281/zenodo.21844857
pi5-thermal-proxy
↗Software-only thermal proxy for commodity SBCs, released with an open 13-hour telemetry dataset intended for reuse.
Cross-validated coupling law, no added sensors
DOI 10.5281/zenodo.21844859
edge-thermal-margin-control
↗Convex thermal-margin allocation for multi-tenant edge inference, with the operating regimes where it fails reported alongside the ones where it wins.
DOI 10.5281/zenodo.21844861
latency-elastic-edge-inference
↗Model-predictive thread allocation that holds a latency service objective while the platform is under thermal and contention pressure.
~3.4x reduction in p99 latency under load
DOI 10.5281/zenodo.21844865
edge-format-tax
↗Matched-byte comparison of GGUF quantization formats on three CPU microarchitectures, with per-token energy from the Pi 5 PMIC. The label on the file turns out to be a poor predictor of what it costs to run.
Matched-byte data and harness on three CPU microarchitectures
DOI 10.5281/zenodo.21938812
edge-breakeven-speedup
↗When multithreading saves energy on an edge CPU: PMIC-measured campaigns on a Raspberry Pi 5 that measure the static power floor at every operating point instead of assuming it.
Static power floor measured at every operating point
DOI 10.5281/zenodo.21987261
thermal-aware-anytime-inference
↗Operating-regime study of thermal-aware anytime (early-exit) inference on a Raspberry Pi 5: when early-exit selection controls temperature, and when it does not. Negative results are shipped with the rest.
DOI 10.5281/zenodo.22163329
gguf-faultscope
↗What one flipped bit does to a block-quantized language model, and where in the GGUF file it has to land to matter.
DOI 10.5281/zenodo.22163327
barrier-spin-crossover
↗Throughput and energy cost of thread-barrier spin-wait thresholds in CPU LLM inference, measured on Cortex-A76 and on a hybrid Alder Lake part.
DOI 10.5281/zenodo.22163331
Tools
Things that outgrew the paper they came from and became usable on their own.
Lab and pipelines
Infrastructure that makes the measurements repeatable.
qgemv-roofline
↗A quantized GEMV kernel ladder measured against its own bandwidth roof: 12 CUDA kernels plus Triton for batch-1 LLM decode, on five NVIDIA GPUs from sm_60 to sm_89.
DOI 10.5281/zenodo.22164007
Edge-sbc-reliability-lab
↗Cross-runtime characterisation harness for machine-learning workloads on single-board computers.
edge-ai-colab-to-pi-pipeline
↗End-to-end path from training a model in Colab to running and measuring it on a Raspberry Pi, built so a result can be re-derived from scratch.
spectral-geometry-instability
↗Research pipeline treating covariance eigenspace rotation as an early risk signal.
DOI 10.5281/zenodo.22163333
Contributions to the infrastructure itself.
Contributions to the projects this research already depends on. Root-cause habits transfer: read the failing system, isolate the fault, fix the cause rather than the symptom.
ai-dynamo/AIPerf
4 merged 8 openThe NVIDIA Dynamo benchmarking tool for LLM inference. Telemetry work from the same measurement habits: host and CPU power collection, NVML energy counters, AMD GPU telemetry, and packaging fixes.
- #1340 test: stop asserting across a Rich line wrap in aggregation output Merged
- #1350 feat(telemetry): probe the NVML energy counter once, fall back cleanly Merged
- #1470 fix(logging): keep log lines a Windows console cannot encode Merged
- #1320 build: make crick an optional dependency Merged
- #1341 feat(telemetry): add a host telemetry collector with a RAPL power path
- #1472 fix(sweep): write pareto_sweep.json and read the requested stat on single-trial sweeps
- #1473 fix(dashboard): show live GPU telemetry after the pane is toggled
- #1474 fix(gpu-telemetry): keep DCGM endpoint credentials out of records, status and logs
- #1437 feat(telemetry): add AMD DME remote GPU telemetry
- #1438 fix(telemetry): read the DCGM hostname label
- #1342 docs(list-metric-aggregation): std uses Welford, not a naive pass
- #1526 docs: clarify that the prefill phase in effective-vs-active metrics is the client-observed TTFT window
vllm-project/vLLM
3 openThe serving engine much of production LLM inference runs on. Fixes and docs from reproducing real failures: tokenizer edge cases, FlashInfer JIT preflight, MoE tuning robustness, WSL2 gotchas.
NVIDIA/garak
3 merged 4 openNVIDIA’s LLM vulnerability scanner. Correctness fixes found by reading the code the way a root-cause engineer reads a failing system.
- #2014 fix(detectors): honour config_root in goodside.RileyIsnt Merged
- #2011 fix: restore unreachable no-default-class error in load_plugin Merged
- #2012 fix(probes): guard badchars ASCII selection against max_ascii bounds Merged
- #2013 fix(detectors): pass response text to the Refusal judge
- #2030 harnesses: skip falsy intents instead of registering them
- #2031 analyze: count prompt characters, not dict length, in count_tokens
- #2032 fix the lite-config hint guard and its masked AttributeError
ArmDeveloperEcosystem/Arm Learning Paths
1 merged 0 openArm's official developer tutorials. A new Learning Path that walks through measuring whether LLM decode is memory-bound on an Arm CPU, with the same roofline method as llama-roofline.
Seeing potholes coming.
A year-long, team-built in-vehicle detection system: stereo depth sensing, CNN detection on embedded hardware, a custom PCB, and a driver alert display. Shipped as a product, not a demo.
Pothole Detection System
ECE senior design · UMass Amherst · Sep 2024 to May 2025Potholes cause billions of dollars in damage a year, and drivers swerving or braking to avoid them is its own hazard. This year-long capstone detects them in real time instead: stereo cameras build a depth point cloud of the road, a CNN detects potholes from the visual and depth data on Raspberry Pi hardware, and an in-vehicle display gives the driver an immediate visual alert. The prototype hit 80 to 90% detection accuracy at speeds up to 35 to 45 mph, with ultrasonic sensors for multi-modal sensing and a custom PCB handling power, sensor interfacing and Pi connectivity. Self-contained, installable in most vehicles, and built to stay affordable.
80-90% detection accuracy, operational at 35-45 mph
Where the hardware habit started.
Award-winning weekend builds from before the research program. Kept here because everything above traces back to them.
WorldWide Rover
HackUMass XII
A rover anyone could drive from anywhere over the internet, with onboard obstacle avoidance and maze solving when nobody was steering.
4Sight
HackUMass X
A wearable that turns ultrasonic distance readings into haptic feedback, so obstacles are felt rather than heard.
Booth Beacon
HackUMass IX
RFID and distance sensors on dining-hall booths, surfaced as a live seat-availability map. First real 3D-printing and Raspberry Pi build.