LLM quantisation formats compared: GGUF, GPTQ, AWQ and what you lose
Four bits is not four bits. The format decides where it runs and what it costs you.
Quantisation makes models fit, and the format you choose decides which hardware can run them, how much quality survives, and whether the file you downloaded will load at all. A practical comparison of the formats you will actually meet.
Quantisation stores a model's weights at lower precision than it was trained in. A 7-billion-parameter model at 16 bits occupies about 14 GB; at 4 bits, about 3.5 GB. That is the difference between needing a datacentre card and running on a laptop.
What the file name does not tell you is that two models both labelled "4-bit" can differ substantially in quality, speed and whether your hardware can load them at all. The format matters as much as the bit count.
What is actually happening
A trained weight is a floating-point number. Quantisation maps a range of those values onto a much smaller set of integers, storing a scale factor so the original can be approximated on the way back out.
Two decisions drive everything that follows:
Granularity. One scale factor for an entire tensor is cheap and crude. One per group of 32, 64 or 128 weights costs a little extra storage and preserves far more. This is why "4-bit" models are never exactly 0.5 bytes per parameter — the group scales add roughly 0.1 to 0.5 bits of overhead, and a model advertised at 4 bits typically lands between 4.2 and 4.8 in practice.
Uniformity. Not all weights matter equally. Formats differ in whether they treat every weight identically or spend more bits on the ones that carry disproportionate influence.
The formats you will meet
GGUF
The format used by llama.cpp and everything built on it. Designed for CPU inference first, with GPU offload as an option — you can place some layers on a GPU and keep the rest in system memory, which is what makes running a large model on modest hardware possible at all.
Its distinguishing feature is the breadth of quantisation levels, labelled in a scheme that rewards learning: Q4_K_M, Q5_K_S, Q8_0 and so on. The number is the nominal bit width; K indicates the k-quant methods, which allocate bits unevenly across a block; S, M and L are small, medium and large variants that trade size against fidelity.
Choose it when you are running on CPU, on Apple silicon, on mixed CPU/GPU, or you simply want the widest choice of size points. It is the most forgiving option and the one with the best tooling for people not running a server.
GPTQ
A GPU-first format that quantises layer by layer, using a small calibration dataset to minimise the error introduced at each step rather than rounding naively.
Because it is calibrated, GPTQ generally preserves quality better than naive rounding at the same bit width. Because it is GPU-first, it expects the whole model in VRAM — there is no graceful CPU fallback.
Choose it when the model fits entirely in VRAM and you want good quality at 4 bits with mature serving support.
AWQ
Activation-aware Weight Quantisation starts from an observation: a small fraction of weights — roughly 1% — disproportionately affect output, and which ones they are can be identified by looking at activation magnitudes rather than the weights themselves.
AWQ protects those salient weights and quantises the rest harder. The practical result is quality close to or better than GPTQ at the same bit width, with inference that is often faster because the format is simpler to decode at runtime.
Choose it when you are serving on GPU and want the best quality-per-bit, particularly at 4 bits.
bitsandbytes (NF4 / INT8)
Quantises on the fly as the model loads, rather than producing a quantised file. That makes it the convenient option — point it at a normal model and set a flag — and the slower one, since nothing is precomputed.
Its real home is training. QLoRA fine-tuning uses NF4 to hold a frozen base model in 4 bits while training adapters on top, which is the technique that put fine-tuning within reach of single consumer cards.
Choose it when you are fine-tuning, or prototyping and do not want to hunt for a pre-quantised file.
What you actually lose
Degradation is not linear, and the useful generalisations are these.
8-bit is close to free. Differences from the full-precision model are hard to detect on most tasks. If 8 bits fits, there is little reason to go lower.
5 to 6 bits is the comfortable band. Small, consistent quality loss that most users will not notice in ordinary use.
4 bits is the interesting decision. Quality is noticeably below the original on demanding tasks — multi-step reasoning, code generation, long-context recall — while remaining perfectly serviceable for summarisation, drafting and conversation. This is where format choice matters most, because a well-implemented 4-bit beats a careless one by a wide margin.
Below 4 bits degrades sharply. 3-bit and 2-bit exist, and are mostly interesting when the alternative is not running the model at all.
One consistent finding across the field is worth internalising: a larger model quantised harder usually beats a smaller model at higher precision, for the same memory budget. A 13B at 4 bits generally outperforms a 7B at 8 bits, and both occupy roughly 7 GB. If you are choosing between them, start with the larger model.
Reading a quantisation label
Faced with a list of files, this is what the names are telling you:
Q8_0— 8-bit, simple scheme. Large, near-lossless.Q6_K— 6-bit k-quant. An excellent default when it fits.Q5_K_M— 5-bit, medium variant. The usual recommendation when 6 is too large.Q4_K_M— 4-bit, medium. The most common choice, and a sensible default.Q4_K_S— 4-bit, small. Noticeably smaller than K_M, noticeably worse.Q3_K_Mand below — use when nothing else fits.
Where both exist and both fit, Q5_K_M over Q4_K_M is usually the better trade; the size difference is modest and the quality difference is real.
The practical decision
- Work out your memory budget — VRAM if everything must fit on the GPU, VRAM plus system memory if you are offloading.
- Pick the largest model that fits at around 4 to 5 bits, rather than a smaller model at higher precision.
- Pick the format by platform: GGUF for CPU, Apple silicon or mixed; AWQ or GPTQ for GPU-only serving; bitsandbytes for fine-tuning.
- Prefer the K-quant medium variants in GGUF unless you are genuinely short of space.
- Test on your own task. Published quality comparisons use benchmarks that may have nothing to do with what you are asking the model to do; a dozen representative prompts at two quantisation levels will tell you more about your use than any leaderboard.
One thing that catches people out
A quantised file is tied to the ecosystem that produced it. A GGUF will not load in a GPTQ-expecting serving stack, and vice versa. Before downloading forty gigabytes, confirm the runtime you intend to use supports that format — this is the single most common reason a correctly downloaded model refuses to start.
