Measured at Concurrency One

1. The claim

A vendor publishes “2.5x faster inference”. You deploy it and see almost nothing, or you see it get worse, and you assume somebody was exaggerating.

Usually nobody was. The number was real on the machine it was measured on, and the machine it was measured on had one request running.

The claim: for inference optimisations, the operating point is part of the result, and reporting the number without it is reporting half a measurement.

2. Why one request is the default

Not conspiracy. Convenience, and it compounds.

Benchmarking a single stream is easy. You need one prompt, one GPU and a stopwatch, and the result is stable and reproducible. Benchmarking at concurrency forty needs a load generator, a request mix, a definition of what you are measuring, a warm-up, and a decision about which percentile matters. One of those is a Tuesday afternoon and the other is a week.

Single-stream numbers are also bigger, which is not usually the reason but never argues against it.

And the audience for the announcement is often running at concurrency one. Somebody with a model on a workstation genuinely does experience the headline figure. For them the number is accurate, and it becomes misleading only when it travels to a team serving traffic.

3. The mechanism, which is the useful part

This is the bit worth actually understanding, because once you have it you can predict the direction yourself instead of taking anyone’s word.

When one request is decoding, the GPU spends most of its time moving weights and cache from memory into compute units that are largely idle. The bottleneck is memory bandwidth. Compute is nearly free, because you have plenty spare.

Add concurrent requests and the same weight read serves many sequences at once. Arithmetic per byte moved climbs. Past some point the compute units are the constraint and memory has headroom, which is the mirror image of where you started.

An optimisation that buys memory traffic at the price of extra compute therefore has its largest effect at concurrency one and its smallest, or a negative one, under load. That is most of them.

4. Two real examples, moving in opposite directions

Speculative decoding, which can invert. A draft model proposes several tokens and the target model verifies them in one pass. At small batch there is idle compute to absorb the verification, so it is close to free, and reported speedups sit around two to three times at well-chosen settings. Raise concurrency and the compute you were borrowing is now the bottleneck, and every rejected draft token is wasted work on a saturated unit. Reported results have speedup dropping below 1.0x at higher concurrency when the configuration was tuned for a single request, and a setup left unreconfigured at a hundred-plus concurrent requests reducing throughput by thirty to forty percent against ordinary decoding.

Note that this is a configuration failure rather than an indictment of the technique. The technique is good. It has an operating range, the range is not printed on the box, and a default chosen at concurrency one is actively harmful outside it.

FP8 KV caching, which shrinks and survives. Here the headline and the production number both appear in the same reporting, which is what good disclosure looks like. For Llama-3.1-8B, a 54 percent reduction in inter-token latency slope at concurrency one becomes a 14.9 percent output throughput increase at concurrency eight.

Figure 2. The same change, the same hardware, two operating points. Roughly a factor of three and a half between the number you would quote and the number you would get. Both are in the source. Only one of them travels.

The reason it survives at all is worth naming, because it is a different mechanism from the one that generated the headline. Halving cache memory lets the scheduler pack more concurrent requests onto the card. So the benefit stops being about decoding each token faster and becomes about serving more of them at once. A related result on 4-bit KV caching keeps around 3.2 times more cache resident in HBM at a fixed memory budget and lifts a cache hit rate from 75.2 to 86.8 percent, delivering roughly double the goodput.

That is the general shape of an optimisation that keeps working under load: the win comes from residency and admission rather than from per-token speed. If a claim is about how fast one token is produced, expect it to fade. If it is about how many requests fit, expect it to hold or grow.

5. Why this is not just a benchmarking complaint

Three ways this costs money rather than just being untidy.

Capacity plans built on single-stream numbers are wrong in the expensive direction. If you sized a fleet assuming a 2.5x speedup that turns out to be 1.15x under your load, you under-provisioned by more than double, and you find out in production.

Defaults ship tuned for the demo. Section 4’s first example is not hypothetical harm. A speculative decoding configuration left at its out-of-the-box setting can cut throughput by a third under real concurrency, and nothing warns you, because from inside the system everything is working exactly as configured.

The comparison between two vendors may be a comparison between two operating points. If one publishes at concurrency one and the other at concurrency thirty-two, the second looks worse while being better for you. This is the same failure as paper 5.20, arriving through the systems door instead of the evaluation one.

6. What to do, and it is genuinely cheap

Ask one question of any performance claim: at what concurrency? If the answer is not in the material, the number is single-stream until proven otherwise. That assumption has been right more often than it has been wrong.

Measure at three points, not one. Your median concurrency, roughly double it, and one request. Three numbers instead of one, and the shape between them tells you which curve in Figure 1 you are on. This is an afternoon with a load generator, and it is the difference between knowing your crossover and discovering it.

Re-tune the knobs that have an operating range. Speculative decoding’s draft length, batch limits and scheduler settings were tuned by somebody with a different workload. At minimum, run with the feature on and off under your own load before believing either.

Prefer optimisations that work through residency. On the evidence in Section 4, the ones that let you fit more work on the card degrade gracefully as you add load, and the ones that make a single stream faster degrade steeply. That is a useful prior when you have to choose without time to measure.

7. The strongest objection

Plenty of real deployments do run at low concurrency. A coding agent on a developer’s machine, a batch job with one worker, an on-premise deployment sized for a handful of internal users: all of these live near concurrency one, and for them the headline number is the right number and this paper is noise.

That is true and it narrows the claim rather than defeating it. The complaint is that single-stream figures get published without their operating point, so a reader cannot tell whether they are in the population the number describes. A number with its conditions attached serves both audiences. A number without them serves whichever one happens to be reading.

8. What this paper does not claim

It does not claim these techniques do not work. Speculative decoding is a genuine advance and FP8 caching pays at both operating points measured.

It does not claim vendors hide the curve. The FP8 example in Section 4 is drawn from reporting that gave both numbers, which is precisely why I could use it.

And the curves in Figure 1 are illustrative. Real speedup-against-concurrency curves depend on model size, hardware, sequence length and scheduler, and I have drawn a shape rather than measurements, which the figure says on its face because a paper about unlabelled operating points should not ship an unlabelled chart.