Q4_K_M vs Q5_K_M vs Q8_0: GGUF Quant Sizes Compared

Q4_K_M and Q5_K_M are two preset recipes in llama.cpp’s quantize tool, and the short answer is a trade of about 0.75 GiB for less rounding error on an 8B model. On Llama 3.1 8B, llama.cpp’s own table puts Q4_K_M at 4.58 GiB (4.89 bits per weight) and Q5_K_M at 5.33 GiB (5.70 bits per weight). Q8_0 is 7.95 GiB.

If Q5_K_M fits in your memory with room left for the context, take it; if it does not, Q4_K_M is the smallest of these presets that llama.cpp’s own perplexity figures put close to the original.

Below are the numbers from the tool’s source and documentation, what the letters in the names mean, and a size calculation you can repeat for other models. Sources were read on November 10, 2026, from the llama.cpp master branch.

The presets side by side

Bits per weight and GiB sizes come from the llama-quantize README ↗. The perplexity column is the label each preset carries in quantize.cpp ↗, measured by the project on Llama 3 8B. Perplexity measures how surprised the model is by a reference text, so a smaller increase means the quantized file behaves more like the original. It is one model on one text, not a ranking of quality for your task. The GB column is our conversion (1 GiB = 1.0737 GB). That conversion is why a “4.9 GB” file on a model page and llama.cpp’s “4.58G” are the same file.

Read the table in steps. Going from Q4_K_M to Q5_K_M costs 16% more memory and cuts the listed perplexity increase by about two thirds. Going from Q5_K_M to Q8_0 costs another 49% for a gap that was already small. Below Q4_K_M the losses grow faster: Q3_K_M saves 0.84 GiB but its listed increase is almost four times larger.

What Q, K, S, M and L mean

The number after Q is the bit width of the main weight blocks. It is not the file’s average.

K marks a k-quant, the family introduced in llama.cpp pull request 1684 ↗. The block layouts in ggml-common.h ↗ show why a “4-bit” type is really 4.5 bits. A Q4_K super-block holds 256 weights in 144 bytes: 128 bytes of 4-bit values, 12 bytes of 6-bit scales and minimums for its eight 32-weight sub-blocks, and two 16-bit super-block scales. 144 × 8 ÷ 256 = 4.5 bits per weight. The same arithmetic gives:

S, M and L (small, medium, large) are mixes. The tool does not quantize every tensor the same way. In llama-quant.cpp ↗, Q4_K_M starts from Q4_K and then raises two tensor groups, attn_v and ffn_down, to Q6_K in selected layers. The selection is the first eighth of the layers, the last eighth, and every third layer in between. On a 32-layer model that is 16 of 32 layers: 0–3, 6, 9, 12, 15, 18, 21, 24, 27 and 28–31. The output tensor goes to Q6_K as well. Q5_K_M follows the same pattern on a Q5_K base. Q4_K_S only bumps a few early layers to Q5_K, which is why it is 0.22 GiB smaller than Q4_K_M.

Q8_0 is not a mix in that sense. It is the older block format: 32 weights stored as 8-bit integers with one 16-bit scale per block.

Checking the 4.58 GiB figure ourselves

The averages above can be rebuilt from tensor shapes. Llama 3.1 8B has 32 layers, a hidden size of 4,096, a feed-forward size of 14,336, 8 key-value heads (1,024-wide K and V projections) and a 128,256-token vocabulary, per its published configuration ↗. That is 8.03 billion weights.

1. All 8.03 billion weights at Q4_K’s 4.5 bits: 4.21 GiB.
2. ffn_down in 16 layers is 940 million weights (11.7% of the model). Moving them from 4.5 to 6.5625 bits adds 0.23 GiB.
3. The output tensor is 525 million weights (6.5%). At Q6_K it adds 0.13 GiB.
4. attn_v in 16 layers is 67 million weights. It adds 0.02 GiB.

Total: 4.58 GiB, or 4.89 bits per weight, which matches llama.cpp’s table. Running the same script for Q5_K_M, Q6_K and Q8_0 gives 5.33, 6.14 and 7.95 GiB, also matching. So Q4_K_M is not a 4-bit file. Almost a fifth of its weights are stored at 6.56 bits, and those are the tensors where the project chose to spend extra bits.

For another model, multiply its parameter count by the bits-per-weight figure and divide by 8. A 14-billion-parameter model at about 4.9 bits comes to roughly 8.6 GB of weights, but treat that as an estimate. Layer shapes change the share of bumped tensors, so check the actual file size on the download page.

Will it fit? Add the context cache

The file is only the weights. The model also keeps a key-value cache for the conversation, and for Llama 3.1 8B at 16-bit precision that is 2 × 32 layers × 1,024 × 2 bytes per token: 128 KiB per token. That is 1 GiB at 8,192 tokens and 4 GiB at 32,768. Our KV cache explainer has the general formula.

An “8 GB” card has 7.45 GiB, and the operating system, display and compute buffers take part of it. On that card Q5_K_M at 8K context leaves about 1 GiB of headroom. Q6_K is tight, and a 32K context does not fit fully on the card at any of these presets without spilling to system RAM. These totals are arithmetic, not measurements on a particular card. The OSBBD memory calculator does the weights side for the models in our catalog. It leaves out the context cache, so add that part yourself.

Names you will see that llama.cpp does not define

The quantize tool’s preset list has no Q4_K_XL or Q4_K_L. Files with those names are uploader-made mixes, usually built with the tool’s per-tensor override options. Read the repository’s notes for what was changed. The IQ types (IQ4_XS, IQ3_M and others) are a separate family, and the tool refuses to make some of the smallest ones (IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS) without an importance matrix: a file of activation statistics gathered from sample text. For downloads, that means two IQ files with the same name can differ depending on the text their uploader used for those statistics.

The container itself, and why a quantized file can still need companion files, is covered in our GGUF files guide. The difference between storage precision and the precision the runtime computes in is in model precision formats.