Skip to content
MN

//Research

Measured, not simulated.

An independent research program on edge and embedded machine-learning systems: 13 papers measured on Arm Cortex-A76 and x86 silicon with hardware performance counters and on-board power instrumentation, 11 currently under review.

Papers
13
Under review
11
Venues
9
Since
Jul 2026

The claim

The memory wall, not compute, governs deep-neural-network inference on real hardware.

One physical idea carried across scales: from mobile CNNs, to Vision Transformers, to on-device language models, measured on Arm and revalidated on x86, and next headed for enterprise servers. Every headline number comes from real silicon, using hardware performance counters and on-board power instrumentation. Each paper reports what the data supports, negative results and the regimes where a method does not help included, and ships the data and harness so any number can be rechecked.

Google Scholar ↗ ORCID 0009-0007-6589-6572 ↗

What the measurements actually look like.

Ten configurations across an Arm SBC and an x86 laptop, plotted from the public artifact. Each machine's measured memory bandwidth draws a ceiling, and decode runs right underneath it on both.

Decode throughput against model size on log-log axes for 10 measured configurations. Points on both platforms sit just below their machine's measured peak memory bandwidth, which is what a memory-bound workload looks like.
i7-12700H peak, 42.1 GB/s Raspberry Pi 5 peak, 13.98 GB/s Qwen 0.5B Q2_K on x86, 8 threads: 100.85 tokens/s at 338.6 MB, 81.1% of peak bandwidth Qwen 0.5B Q4_K_M on x86, 8 threads: 85.64 tokens/s at 397.8 MB, 80.9% of peak bandwidth Qwen 0.5B Q8 on x86, 8 threads: 76.08 tokens/s at 531.1 MB, 96% of peak bandwidth Llama 1B Q4_K_M on x86, 14 threads: 48.13 tokens/s at 807.7 MB, 92.3% of peak bandwidth Qwen 1.5B Q4_K_M on x86, 14 threads: 37.65 tokens/s at 986 MB, 88.2% of peak bandwidth Qwen 3B Q4_K_M on x86, 14 threads: 20.35 tokens/s at 1929.9 MB, 93.3% of peak bandwidth Qwen 7B Q4_K_M on x86, 14 threads: 9.62 tokens/s at 4683.1 MB, 107% of peak bandwidth Qwen 0.5B Q4_K_M on Arm, 3 threads: 28.05 tokens/s at 397.8 MB, 79.8% of peak bandwidth Llama 1B Q4_K_M on Arm, 2 threads: 14.41 tokens/s at 807.7 MB, 83.3% of peak bandwidth Qwen 1.5B Q4_K_M on Arm, 2 threads: 12.2 tokens/s at 986 MB, 86% of peak bandwidth 500 MB1 GB2 GB5 GB 102550100 Model size on disk Decode, tokens/s

Ten measured configurations, 339 MB to 4.7 GB, each at its best thread count. The dashed lines are each machine's measured peak memory read bandwidth, not a spec sheet figure. Decode lands at 80 to 86 percent of peak on the Pi and 81 to 96 percent on the laptop. The 7B point reads at 107 percent, which means the peak benchmark slightly understates what the memory system delivers under this access pattern, and it is left in rather than trimmed. Throughput here is set by how fast the machine can read the weights, not by how fast it can multiply.

A light engineering-grid notebook card: latency spikes at the first inference after a device wakes, 5 to 23 times slower than every one after it, then settles to steady state. A light engineering-grid notebook card: four small trace rows, one per shipped default that turned out to be quietly wrong, captioned 2026 edge inference measurement notebook.

Two of the findings, as they left the bench

Publications · 13

  1. 01

    The Memory Wall Governs Edge DNN Inference

    Thread scaling, multi-tenant interference and INT8 speedups are not three separate phenomena. All three fall out of a single measured bandwidth ceiling.

    20.5% leave-one-model-out prediction error across 9 CNNs and a ViT

    RooflineMemory bandwidthCortex-A76

    ACM Transactions on Embedded Computing Systems

    Journal

    Under review

    Submitted 28 Jul 2026

  2. 02

    The Cold-Start Tax

    The first systematic characterisation of the duty-cycled wake transient on an edge SBC. The tax is weight-loading-bound, predictable from model size alone, and falls off an eviction cliff. Includes GD-Tax, a tax-aware cache that beats LRU and LFU.

    5x to 23x wake tax, R^2 = 1.00 loading-bound fit, >80x blow-up past eviction

    Duty cyclingCache policyGD-Tax
    › Cite 
    @software{jacob_cold_start_2026,
  author = {Jacob, Manu Nicholas},
  title  = {The Cold-Start Tax},
  year   = {2026},
  doi    = {10.5281/zenodo.21844857},
  url    = {https://doi.org/10.5281/zenodo.21844857}
}

    IEEE Internet of Things Journal

    Journal

    Under review

    Submitted 6 Aug 2026

    Artifact

    edge-cold-start-tax ↗ 10.5281/zenodo.21844857
  3. 03

    The Narrow Regime of Thermal-Aware Anytime Inference

    Anytime inference is widely proposed for thermally constrained edge devices. This maps the narrow regime in which exit selection affects temperature at all, and shows quantization compresses the exit ladder enough to shrink the authority of the controller.

    Quantization compresses the exit ladder from 20.6x to 6.9x

    Anytime inferenceSLOThermal

    Venue in selection

    Journal

    In rebuild

    Rebuilt around a measured state-of-the-art baseline

  4. 04

    Four Walls Before Hello: Deploying LLMs at the Edge

    A deployment guide rather than a new measurement: the four limits that decide whether an on-device language model is viable at all, drawn together from the bandwidth, cold-start, quantization and energy results and written for people shipping hardware.

    Synthesises four measurement papers into one deployment guide

    LLM inferenceEdge deploymentInvited special issue

    IEEE Design & Test

    Journal

    Under review

    Submitted 17 Aug 2026

  5. 05

    The Cold-Start Tax at the Edge of Language

    The wake transient for on-device LLMs, where the model is 50 to 100x larger than a CNN. A phase-level decomposition shows about a fifth of the tax is the loader rather than the storage device, and a purpose-built loader recovers it.

    A fifth of the LLM wake tax is software, and a better loader wins it back

    LLM inferenceDuty cyclingStorage

    ACM Transactions on Embedded Computing Systems

    Journal

    Under review

    Submitted 17 Aug 2026

  6. 06

    When Does INT8 Actually Pay on an Edge CPU?

    Post-training INT8 measured across three microarchitectures on Arm and x86. The dominant effect is dynamic versus static quantization, worth about a factor of four; once both are produced statically, the export representation everyone argues about is worth a median of 1.5% across 27 model and core pairs.

    Dynamic vs static: about 4x. The representation: a median 1.5%

    QuantizationCross-platformONNX Runtime

    IEEE Embedded Systems Letters

    Letters

    Under review

    Submitted 17 Aug 2026

  7. 07

    CPU Utilization as a Software-Only Thermal Proxy

    A cross-validated CPU-temperature coupling law that needs no extra sensors, released with an open 13-hour telemetry dataset so others can re-run it.

    Open 13-hour telemetry dataset, no added hardware

    ThermalTelemetryOpen dataset
    › Cite 
    @software{jacob_thermal_proxy_2026,
  author = {Jacob, Manu Nicholas},
  title  = {CPU Utilization as a Software-Only Thermal Proxy},
  year   = {2026},
  doi    = {10.5281/zenodo.21844859},
  url    = {https://doi.org/10.5281/zenodo.21844859}
}

    IEEE Embedded Systems Letters

    Letters

    Under review

    Revised and resubmitted 5 Sep 2026

    Artifact

    pi5-thermal-proxy ↗ 10.5281/zenodo.21844859
  8. 08

    The Break-Even Parallel Speedup

    Running inference on every core is the default on edge CPUs. Whether it saves energy turns on the parallel speedup against the power ratio, and on whether the static power of the board is charged to the inference. Measured on a Raspberry Pi 5 with the static floor measured at every operating point.

    Break-even depends on what you charge to the inference

    EnergyMultithreadingParallelism
    › Cite 
    @software{jacob_break_even_2026,
  author = {Jacob, Manu Nicholas},
  title  = {The Break-Even Parallel Speedup},
  year   = {2026},
  doi    = {10.5281/zenodo.21987261},
  url    = {https://doi.org/10.5281/zenodo.21987261}
}

    IEEE Transactions on Computers

    Journal

    Under review

    Resubmitted as a regular paper 6 Sep 2026

    Artifact

    edge-breakeven-speedup ↗ 10.5281/zenodo.21987261
  9. 09

    One Law, Two Memory Systems

    The bandwidth ceiling that governs CNN inference governs on-device LLM decode too, and the KV cache turns it into a hard capacity wall. The same law holds on x86 at three times the bandwidth, so the same one-line roofline transfers between the two CPUs measured.

    Decode roofline R^2 = 0.98, unchanged on x86 at three times the bandwidth

    LLM inferenceKV cachellama.cpp
    › Cite 
    @software{jacob_edge_llm_2026,
  author = {Jacob, Manu Nicholas},
  title  = {One Law, Two Memory Systems},
  year   = {2026},
  doi    = {10.5281/zenodo.21844855},
  url    = {https://doi.org/10.5281/zenodo.21844855}
}

    IEEE Internet of Things Journal

    Journal

    Under review

    Rebuilt after the TC decline, submitted 22 Sep 2026

    Artifact

    edge-llm-memory-wall ↗ 10.5281/zenodo.21844855
  10. 10

    Your Quantization Format Is Not Free

    Controlled measurements of GGUF quantization formats on three CPU microarchitectures. At small model sizes a format label predicts poorly what the file actually contains, and the community-default format pays a measurable cost. Full numbers after the review concludes.

    What a format label does and does not tell you

    QuantizationGGUFEnergy
    › Cite 
    @software{jacob_format_tax_2027,
  author = {Jacob, Manu Nicholas},
  title  = {Your Quantization Format Is Not Free},
  year   = {2027},
  doi    = {10.5281/zenodo.21938812},
  url    = {https://doi.org/10.5281/zenodo.21938812}
}

    HotMobile 2027

    Workshop

    Under review

    Submitted 22 Sep 2026

    Artifact

    edge-format-tax ↗ 10.5281/zenodo.21938812
  11. 11

    Use More Threads, Not Fewer

    LLM decode thread scaling measured on a hybrid-core i7-12700H and a Raspberry Pi 5. Oversubscription costs predictability rather than speed, and a controlled headroom experiment names the cause. Extends and revises the Letter-length version a reviewer asked to see grown up.

    Oversubscription costs predictability, not throughput

    LLM inferenceHybrid coresScheduling

    IEEE Trans. on Parallel and Distributed Systems

    Journal

    Under review

    Extended after the CAL decline, submitted 23 Sep 2026

  12. 12

    When Thermal-Margin Control Helps and When It Hurts

    An honest operating-regime study of convex thermal allocation for multi-tenant edge inference, including the regimes where the controller is the wrong answer.

    Maps the regime boundary rather than claiming a universal win

    ThermalControlMulti-tenant
    › Cite 
    @software{jacob_thermal_margin_2026,
  author = {Jacob, Manu Nicholas},
  title  = {When Thermal-Margin Control Helps and When It Hurts},
  year   = {2026},
  doi    = {10.5281/zenodo.21844861},
  url    = {https://doi.org/10.5281/zenodo.21844861}
}

    Venue in selection

    Journal

    Being re-homed

    Rebuilt with matched baselines; returned on scope 10 Oct 2026

    Artifact

    edge-thermal-margin-control ↗ 10.5281/zenodo.21844861
  13. 13

    Latency-Optimal Is Not Energy-Optimal

    Running an edge SBC at maximum clock wastes roughly a fifth of the energy per inference, and the waste is core-stall energy rather than DRAM energy. That reverses the usual intuition.

    ~20% energy wasted at max clock, with a deployable clock-selection policy

    EnergyDVFSPower telemetry

    Sustainable Computing: Informatics and Systems

    Journal

    Under review

    Declined at ACM TECS, resubmitted 27 Sep 2026

Statuses are kept current by hand and papers are listed in submission order.

Next The other side of the desk: reviewing