Skip to content
MN

//Research

Measured, not simulated.

An independent research program on edge and embedded machine-learning systems: 13 papers measured on Arm Cortex-A76 and x86 silicon with hardware performance counters and on-board power instrumentation, 11 currently under review.

Papers
13
Under review
11
Venues
7
Since
Jul 2026

The claim

The memory wall, not compute, governs deep-neural-network inference on real hardware.

One physical idea carried across scales: from mobile CNNs, to Vision Transformers, to on-device language models, measured on Arm and revalidated on x86, and next headed for enterprise servers. Every headline number comes from real silicon, using hardware performance counters and on-board power instrumentation. Where a claim failed to reproduce, the paper was rewritten to whatever the data actually supported, negative results included, and two early drafts that presented simulation as measurement were discarded outright. That standard keeps costing claims, which is the point.

Google Scholar ↗ ORCID 0009-0007-6589-6572 ↗

What the measurements actually look like.

Ten configurations across an Arm SBC and an x86 laptop, plotted from the public artifact. Each machine's measured memory bandwidth draws a ceiling, and decode runs right underneath it on both.

Decode throughput against model size on log-log axes for 10 measured configurations. Points on both platforms sit just below their machine's measured peak memory bandwidth, which is what a memory-bound workload looks like.
i7-12700H peak, 42.1 GB/s Raspberry Pi 5 peak, 13.98 GB/s Qwen 0.5B Q2_K on x86, 8 threads: 100.85 tokens/s at 338.6 MB, 81.1% of peak bandwidth Qwen 0.5B Q4_K_M on x86, 8 threads: 85.64 tokens/s at 397.8 MB, 80.9% of peak bandwidth Qwen 0.5B Q8 on x86, 8 threads: 76.08 tokens/s at 531.1 MB, 96% of peak bandwidth Llama 1B Q4_K_M on x86, 14 threads: 48.13 tokens/s at 807.7 MB, 92.3% of peak bandwidth Qwen 1.5B Q4_K_M on x86, 14 threads: 37.65 tokens/s at 986 MB, 88.2% of peak bandwidth Qwen 3B Q4_K_M on x86, 14 threads: 20.35 tokens/s at 1929.9 MB, 93.3% of peak bandwidth Qwen 7B Q4_K_M on x86, 14 threads: 9.62 tokens/s at 4683.1 MB, 107% of peak bandwidth Qwen 0.5B Q4_K_M on Arm, 3 threads: 28.05 tokens/s at 397.8 MB, 79.8% of peak bandwidth Llama 1B Q4_K_M on Arm, 2 threads: 14.41 tokens/s at 807.7 MB, 83.3% of peak bandwidth Qwen 1.5B Q4_K_M on Arm, 2 threads: 12.2 tokens/s at 986 MB, 86% of peak bandwidth 500 MB1 GB2 GB5 GB 102550100 Model size on disk Decode, tokens/s

Ten measured configurations, 339 MB to 4.7 GB, each at its best thread count. The dashed lines are each machine's measured peak memory read bandwidth, not a spec sheet figure. Decode lands at 80 to 86 percent of peak on the Pi and 81 to 96 percent on the laptop. The 7B point reads at 107 percent, which means the peak benchmark slightly understates what the memory system delivers under this access pattern, and it is left in rather than trimmed. Throughput here is set by how fast the machine can read the weights, not by how fast it can multiply.

Publications · 13

  1. 01

    CPU Utilization as a Software-Only Thermal Proxy

    A cross-validated CPU-temperature coupling law that needs no extra sensors, released with an open 13-hour telemetry dataset so others can re-run it.

    Open 13-hour telemetry dataset, no added hardware

    ThermalTelemetryOpen dataset
    › Cite 
    @software{jacob_thermal_proxy_2026,
  author = {Jacob, Manu Nicholas},
  title  = {CPU Utilization as a Software-Only Thermal Proxy},
  year   = {2026},
  doi    = {10.5281/zenodo.21844859},
  url    = {https://doi.org/10.5281/zenodo.21844859}
}

    IEEE Embedded Systems Letters

    Letters

    Under review

    Submitted 27 Jul 2026

    Artifact

    pi5-thermal-proxy ↗ 10.5281/zenodo.21844859
  2. 02

    The Memory Wall Governs Edge DNN Inference

    Thread scaling, multi-tenant interference and INT8 speedups are not three separate phenomena. All three fall out of a single measured bandwidth ceiling.

    20.5% leave-one-model-out prediction error across 9 CNNs and a ViT

    RooflineMemory bandwidthCortex-A76

    ACM Transactions on Embedded Computing Systems

    Journal

    Under review

    Submitted 28 Jul 2026

  3. 03

    Latency-Optimal Is Not Energy-Optimal

    Running an edge SBC at maximum clock wastes roughly a fifth of the energy per inference, and the waste is core-stall energy rather than DRAM energy. That reverses the usual intuition.

    ~20% energy wasted at max clock, with a deployable clock-selection policy

    EnergyDVFSPower telemetry

    ACM Transactions on Embedded Computing Systems

    Journal

    Under review

    Submitted 6 Aug 2026

  4. 04

    When Thermal-Margin Control Helps and When It Hurts

    An honest operating-regime study of convex thermal allocation for multi-tenant edge inference, including the regimes where the controller is the wrong answer.

    Maps the regime boundary rather than claiming a universal win

    ThermalControlMulti-tenant
    › Cite 
    @software{jacob_thermal_margin_2026,
  author = {Jacob, Manu Nicholas},
  title  = {When Thermal-Margin Control Helps and When It Hurts},
  year   = {2026},
  doi    = {10.5281/zenodo.21844861},
  url    = {https://doi.org/10.5281/zenodo.21844861}
}

    IEEE Embedded Systems Letters

    Letters

    Under review

    Submitted 6 Aug 2026

    Artifact

    edge-thermal-margin-control ↗ 10.5281/zenodo.21844861
  5. 05

    The Cold-Start Tax

    The first systematic characterisation of the duty-cycled wake transient on an edge SBC. The tax is weight-loading-bound, predictable from model size alone, and falls off an eviction cliff. Includes GD-Tax, a tax-aware cache that beats LRU and LFU.

    5x to 23x wake tax, R^2 = 1.00 loading-bound fit, >80x blow-up past eviction

    Duty cyclingCache policyGD-Tax
    › Cite 
    @software{jacob_cold_start_2026,
  author = {Jacob, Manu Nicholas},
  title  = {The Cold-Start Tax},
  year   = {2026},
  doi    = {10.5281/zenodo.21844857},
  url    = {https://doi.org/10.5281/zenodo.21844857}
}

    IEEE Internet of Things Journal

    Journal

    Under review

    Submitted 6 Aug 2026

    Artifact

    edge-cold-start-tax ↗ 10.5281/zenodo.21844857
  6. 06

    The Memory Wall at the Edge of Language

    The bandwidth ceiling that governs CNN inference governs on-device LLM decode too, and the KV cache turns it into a hard capacity wall. The same law holds on x86 at three times the bandwidth, so the roofline is the platform-independent part.

    Decode roofline R^2 = 0.98, unchanged on x86 at three times the bandwidth

    LLM inferenceKV cachellama.cpp
    › Cite 
    @software{jacob_edge_llm_2026,
  author = {Jacob, Manu Nicholas},
  title  = {The Memory Wall at the Edge of Language},
  year   = {2026},
  doi    = {10.5281/zenodo.21844855},
  url    = {https://doi.org/10.5281/zenodo.21844855}
}

    IEEE Transactions on Computers

    Journal

    Under review

    Submitted 7 Aug 2026

    Artifact

    edge-llm-memory-wall ↗ 10.5281/zenodo.21844855
  7. 07

    The Narrow Regime of Thermal-Aware Anytime Inference

    Anytime inference is widely proposed for thermally constrained edge devices. This maps the narrow regime where it actually pays, and shows quantization compresses the exit ladder enough to shrink the controller’s authority.

    Quantization compresses the exit ladder from 20.6x to 6.9x

    Anytime inferenceSLOThermal

    IEEE Embedded Systems Letters

    Letters

    Ready to submit

    Withdrawn from DATE, recut for IEEE ESL

  8. 08

    The Hybrid-Core Decode Cliff

    On CPUs that mix performance and efficiency cores, LLM decode throughput does not scale smoothly with thread count. It collapses once the scheduler starts placing decode work on the efficiency cores, and the fastest configuration is not the widest one.

    Decode throughput collapses on performance-plus-efficiency CPUs

    LLM inferenceHybrid coresScheduling

    IEEE Computer Architecture Letters

    Letters

    Under review

    Submitted 17 Aug 2026

  9. 09

    Four Walls Before Hello: Deploying LLMs at the Edge

    A deployment guide rather than a new measurement: the four limits that decide whether an on-device language model is viable at all, drawn together from the bandwidth, cold-start, quantization and energy results and written for people shipping hardware.

    Synthesises four measurement papers into one deployment guide

    LLM inferenceEdge deploymentInvited special issue

    IEEE Design & Test

    Journal

    Under review

    Submitted 17 Aug 2026

  10. 10

    The Cold-Start Tax at the Edge of Language

    The wake transient for on-device LLMs, where the model is 50 to 100x larger than a CNN. A phase-level decomposition shows about a fifth of the tax is the loader rather than the storage device, and a purpose-built loader recovers it.

    A fifth of the LLM wake tax is software, and a better loader wins it back

    LLM inferenceDuty cyclingStorage

    ACM Transactions on Embedded Computing Systems

    Journal

    Under review

    Submitted 17 Aug 2026

  11. 11

    When Does INT8 Actually Pay on an Edge CPU?

    Post-training INT8 measured across three microarchitectures on Arm and x86. The dominant effect is dynamic versus static quantization, worth more than 4x; the export representation everyone argues about is worth about 18%. Grew out of correcting our own earlier misattribution.

    Dynamic vs static is worth >4x; the representation, ~18%

    QuantizationCross-platformCorrection

    IEEE Embedded Systems Letters

    Letters

    Under review

    Submitted 17 Aug 2026

  12. 12

    The Break-Even Parallel Speedup

    For early-exit inference, the latency-optimal configuration costs about twice the energy of the energy-optimal one at shallow exits, and the first-order lever is thread count, not clock. A closed-form break-even rule predicts the right choice.

    Rule validated on 64 of 64 measured configurations

    EnergyEarly exitParallelism
    › Cite 
    @software{jacob_break_even_2026,
  author = {Jacob, Manu Nicholas},
  title  = {The Break-Even Parallel Speedup},
  year   = {2026},
  doi    = {10.5281/zenodo.21987261},
  url    = {https://doi.org/10.5281/zenodo.21987261}
}

    IEEE Computer Architecture Letters

    Letters

    Under review

    Submitted 17 Aug 2026

    Artifact

    edge-breakeven-speedup ↗ 10.5281/zenodo.21987261
  13. 13

    Profiling Inference on a Consumer Laptop GPU

    A clock-gated methodology that makes consumer-GPU inference numbers reproducible between runs, kept honest by a drift artifact we caught in our own background jobs and left in as the counterexample.

    Clock-gated protocol for repeatable GPU measurement

    GPUMethodologyReproducibility

    EuroMLSys (EuroSys workshop)

    Workshop

    Ready, awaiting CFP

    CFP expected Nov 2026

Statuses are kept current by hand and papers are listed in submission order. Work that failed to reproduce under re-examination was withdrawn rather than listed, and corrections to our own published claims are recorded in the papers that supersede them.