Kubernetes did not slow my benchmark down. One line of YAML took 64 percent.

A lot of language model benchmarking now happens inside Kubernetes, because that is where the GPUs are and that is what the platform team hands you. I could not find anyone who had measured what the pod does to the numbers that come out.

I had assumed there was a tax. Some percentage you pay for the namespaces, the overlay filesystem, the container network, the extra layer of scheduler. Small, probably, but there. I moved my benchmark harness into a pod partly to find out what it was.

I could not find it.

What I actually ran

Five arms, one laptop, an Intel i7-12700H. A plain process on the Linux host, and then the same work in a Kubernetes pod four times over: with no CPU limit at all, with a limit of eight cores, with four, and with two. The benchmark asked for four threads in every single one.

Three things had to be true or none of the rest would mean anything.

The same binary in every arm. llama.cpp is built once, inside the container image, and the copy the host arm runs is pulled back out of that same image afterwards. If each side had compiled its own, I would have been measuring a compiler.

The same file on disk. The pods mount the model directory off the node, read only, so every arm reads the same inode. No copy of the weights sits inside an overlay filesystem where it could quietly change the storage path.

One arm at a time. This is the one I nearly got wrong. Those five arms are five entries in a config file but they are one CPU, and if the scheduler runs them together it is not measuring containers, it is measuring them fighting each other. They share a resource group and take turns. When I first wrote the turn taking it was first come first served, which in practice means whichever worker grabbed the lock after finishing, so one arm ran its entire queue before any other arm started and each arm sat in a different part of the afternoon’s thermal history. Round robin fixed it.

Then the whole matrix twice, on separate passes, so drift over the sweep is visible rather than assumed away.

Decode throughput in a pod, relative to an ordinary process on the same machine. Fifty runs, two independent passes, five model and quantization combinations, four threads throughout. Each dot is one model. The shaded band is the width this measurement can actually resolve, so a bar inside it is a bar you are not entitled to read anything into. Data: ml-systems-lab.

Three bars sit inside the noise

The pod with no limit, the pod limited to eight cores and the pod limited to four all land within about three percent of the bare process, and every one of those three percents is smaller than the run to run spread of the measurement that produced it. Fourteen of the fifteen individual comparisons cannot be separated from noise at all.

I want to be careful about what that sentence says. It does not say containerisation is free. It says that on this machine, with this workload, at this precision, I could not find the cost. Those are different claims and the second one is the only one I measured. If the tax is one percent it is under my floor and it will stay there until somebody runs this on quieter hardware.

The direction is mildly funny, though. All three came out marginally on the pod’s side rather than the host’s. I have no story for that and I do not think there is one to tell at that size.

The fourth bar is not about containers

Two cores against four threads costs 64 percent of throughput, and unlike everything above, that one is resolved on every single point. There is no ambiguity in it at all.

The container has nothing to do with it. You are asking for more parallelism than the control group will let you have, and Linux enforces a CPU limit by handing the cgroup a budget of runtime every hundred milliseconds and stopping it when the budget is gone, so four threads against a two core quota spend a chunk of every period frozen, waiting for the next one. The container is incidental. You would get the same behaviour from a bare cgroup, and the same from docker run --cpus=2.

What makes it worth writing down is how easy it is to arrive at by accident. The thread count usually comes from a default deep inside the inference library, or from whatever nproc reported on the machine where somebody wrote the manifest. The CPU limit comes from a completely separate conversation about capacity planning, held by different people, months earlier. Nothing in Kubernetes connects the two or warns you when they disagree. The pod runs. It just runs at a third of the speed, and the number lands in a spreadsheet next to numbers taken on other hardware.

There is a smaller sting in the tail. A pod squeezed that hard is also hard to kubectl exec into, because the control path shares the quota with the workload. Three attempts on that arm failed outright mid sweep and the harness had to open a circuit breaker on it and come back later. So the arm that is misconfigured is also the arm that is awkward to go and inspect.

The ranking survived, which is the useful part

I ran five different model and quantization combinations through all of this. In every arm, including the one losing two thirds of its throughput, they came out in the same order.

That is the answer most people actually need. If you are benchmarking inside a pod to decide between two quantization formats, or two model sizes, the pod does not change your winner. The cost, where there is one, is close enough to multiplicative that the comparison rides through it.

I did not expect to write that sentence, because my first pass at the analysis said the opposite. It reported that four of four arms reordered the models, and I nearly believed it. The reorderings were all swaps between models a few percent apart, in data whose worst run to run spreads were over twenty percent. It was noise wearing a conclusion. The analysis now refuses to call any difference real unless it clears two standard errors, which is why the chart has a shaded band on it and why three of the four bars are drawn as not-a-finding.

That correction is the reason I trust the rest of it.

What I would do with this

Check that your CPU limit is at least your thread count. It is one line in a manifest against one setting in a config file, they are usually written by different people, and the failure is silent.

If you are comparing options rather than quoting absolute numbers, a pod is fine. The ranking held here through a 64 percent penalty.

If you are quoting absolute throughput from inside a pod, put the quota next to it. A number with no quota beside it cannot be compared with anything.

The honest limits

The host arm is a process on a WSL2 Linux host, which is itself a virtual machine. That layer is in all five arms so it cancels out of every comparison here, but these are not bare metal numbers and I am not going to call them that.

One machine, one CPU architecture, one inference engine, one model family. This is throughput, not tail latency, and tail latency under a CFS quota is a nastier question, because throttling lands on individual requests rather than on an average. And a home lab is not a fleet.

The framework, the fifty records, the report and the figure are in ml-systems-lab, archived at 10.5281/zenodo.22162103. The whole thing is one config file:

git clone https://github.com/manunicholasjacob/ml-systems-lab
cd ml-systems-lab && pip install -e .
# docker/README.md is the setup: build the image, load it into k3s, pull the
# host arm's binaries back out of that same image, stage the models on the node
mlsys run configs/containerization.yaml --concurrent
python tools/containerization_report.py runs/containerization-tax --out /tmp/ct

What I would like is this on a machine that is not mine. A server chip with more cores, where a quota is a likelier thing to be running under, or Arm, where the memory system behaves differently enough that the answer might not be the same one. If you run it, send me the numbers.