Why your local LLM is slow: memory bandwidth, not TFLOPS
Token generation is a memory-bound workload. The spec that predicts it is the one nobody quotes.
A GPU with twice the compute can generate tokens no faster than one with half of it. Generation reads the entire model from memory for every token produced, so bandwidth sets the ceiling — and you can calculate yours before you buy anything.
People buy a faster GPU, run the same model, and find generation barely improved. The usual explanation — drivers, framework, thermal throttling — is almost always wrong. The real reason is that single-stream token generation is not a compute problem at all.
What happens when a model generates one token
To produce a single token, the model performs a forward pass. Every weight participates. On a GPU, those weights live in VRAM and must be moved into the compute units to be used.
The arithmetic per weight is trivial — a multiply and an accumulate. The movement is not. For a single token with a batch size of one, each weight is read, used for roughly two floating-point operations, and discarded. The ratio of computation to data movement is about as bad as it gets.
That ratio has a name: arithmetic intensity. When it is low, the processor spends most of its time waiting for memory, and adding compute changes nothing. Token generation sits firmly in that regime.
The calculation that predicts your ceiling
Because every weight is read once per token, the upper bound follows directly:
max tokens/sec = memory bandwidth (GB/s) ÷ model size in memory (GB)Work an example. A 7-billion-parameter model quantised to 4 bits occupies roughly 3.5 GB. On hardware with 900 GB/s of bandwidth:
900 ÷ 3.5 ≈ 257 tokens/secYou will not see 257. Attention, sampling, framework overhead and imperfect memory access patterns all take a cut, and 50–70% of theoretical is a realistic band. But the number tells you the shape of the answer, and more usefully it tells you the ratio between two machines you are choosing between.
Run the same model on a machine with 100 GB/s and the ceiling is about 28 tokens/sec. No amount of compute closes that gap, because compute was never the constraint.
Why this explains Apple silicon
Apple's unified memory confuses people because it performs unlike anything else in two opposite directions at once.
Capacity is exceptional: the GPU addresses the whole system memory pool, so a machine with 64 GB or 128 GB can hold models that would otherwise demand datacentre hardware. A 70B model at 4-bit needs roughly 40 GB and simply fits.
Bandwidth is the other half of the story, and it varies enormously within the product line — the base chip, the Pro, the Max and the Ultra are not close to one another. Two machines with identical memory capacity can differ by a factor of several in generation speed, because the tier determines the memory interface width. When comparing Apple machines for inference, the capacity number tells you what fits and the bandwidth number tells you how fast it runs. Both are published; only one is usually quoted.
Why CPUs are slower than their core count suggests
The same formula explains CPU inference. System memory bandwidth on a typical desktop is an order of magnitude below a discrete GPU's. Adding cores does not help a memory-bound workload — the cores are already waiting.
It also explains why memory channels matter more than clock speed for this task. A dual-channel configuration has roughly twice the bandwidth of single-channel at the same speed, and that difference shows up almost linearly in tokens per second. If you are running models on a CPU and have one memory stick, adding a second matching stick is usually the cheapest meaningful upgrade available.
Why quantisation speeds things up
Quantisation is usually explained as a way to save memory. It is, but the speed effect follows from the same equation and is often the bigger practical win.
Halving the bytes per parameter halves the data read per token, which doubles the ceiling:
7B model, 900 GB/s of bandwidth
FP16 14 GB 900 ÷ 14 ≈ 64 tok/s
INT8 7 GB 900 ÷ 7 ≈ 128 tok/s
INT4 3.5 GB 900 ÷ 3.5 ≈ 257 tok/sThis is why a quantised model on modest hardware can outrun a full-precision model on much better hardware. It is not an optimisation in the usual sense — it is a direct consequence of moving less data.
Prompt processing is the opposite problem
Everything above describes generation. Reading your prompt behaves completely differently, and conflating the two causes a lot of confused benchmarking.
When the model ingests a prompt, it processes many tokens at once. The weights are read once and reused across all of them, so arithmetic intensity is high and the workload becomes compute-bound — the regime where TFLOPS finally matter.
This is why a long document can take a noticeable pause before the first token appears, and then stream at a steady rate afterwards. Two different bottlenecks, one after the other:
- Time to first token — dominated by compute, scales with prompt length.
- Tokens per second after that — dominated by bandwidth, roughly constant.
A benchmark quoting a single number has usually measured one and ignored the other. When comparing hardware, ask which one is being reported.
Batching changes the regime
If you serve several requests concurrently, the weights read for one sequence are reused for the others. Arithmetic intensity rises, and the workload shifts from memory-bound towards compute-bound.
The practical consequence is that total throughput across many users can scale well with batch size even though each individual user's generation speed does not improve. A server handling ten conversations does more total work per second than one handling a single conversation, on the same hardware.
It also means benchmarks from serving infrastructure do not transfer to a single-user desktop. A figure measured at batch 64 describes a different regime from the one you will experience alone.
How to use this when buying
- Find the memory bandwidth of each candidate. It is published for every GPU, every Apple chip tier, and derivable for a CPU from its memory type and channel count.
- Work out the model's size in memory at the quantisation you intend to use — parameters × bytes per parameter.
- Divide. That is your ceiling. Expect to achieve somewhere between half and two thirds of it.
- Check capacity second. A fast machine that cannot hold the model is useless; a slow one that can is merely slow.
- Only then look at compute — and only if your workload is prompt-heavy, fine-tuning, or serving many users at once.
The exceptions worth knowing
Mixture-of-experts models break the simple formula. Only a subset of parameters activates per token, so the data read is far smaller than the total model size — which is why an MoE model with a large parameter count can generate faster than a dense model with fewer parameters. Use the active parameter count in the calculation, not the total.
Speculative decoding uses a small draft model to propose several tokens that the large model then verifies in one pass. Because verification is a batched operation, it moves part of the work into the compute-bound regime and can exceed the naive bandwidth ceiling.
Both are real, both are increasingly common, and both are reasons to treat the formula as a floor on your understanding rather than a hard limit on the hardware.
The short version
Generation speed is bandwidth divided by model size. Capacity decides what runs at all. Compute matters for prompts, training and serving many users — not for watching tokens appear one at a time. Check the bandwidth figure before the TFLOPS figure, because only one of them predicts the thing you are about to experience.
