A GGUF file is a single binary file that holds a model’s weights together with the metadata a local runtime needs to load them, such as the architecture, the tokenizer and how each tensor is encoded. GGUF files are what llama.cpp and other tools in the GGML ecosystem load. Before downloading one, check four things the .gguf extension does not tell you: which model it contains, how its weights are encoded, whether it needs companion files, and whether your software supports that architecture.
Last checked 15 November 2026 against the GGUF specification, the Hugging Face GGUF documentation and the file lists of the repositories named below.
A download page may offer dozens of files for one model. Choosing the largest is not automatically the best choice, and choosing the smallest can leave you with a projector, an adapter or one shard rather than a usable model. Here is how to read that list before spending bandwidth.
What a GGUF file contains
Hugging Face’s GGUF documentation describes the format as a binary container for tensors and metadata. Tensors hold the numerical arrays used by the model; metadata describes how the runtime should interpret them. Hugging Face’s file viewer can show tensor names, shapes and precision without making you download the entire file.
The GGUF specification defines the header precisely. A valid file starts with the four bytes GGUF, followed by a format version (the current specification requires version 3), the number of tensors and the number of metadata key-value pairs. Model details such as general.architecture, general.basename and general.size_label live in that metadata, not in the filename.
That split matters: a program may read GGUF bytes correctly and still lack support for a particular model architecture or feature. GGUF describes a file format, not a promise that every local AI application can run every file.
Read the name, then check the model card
The specification proposes this naming pattern, with each part separated by a hyphen:
[<Sidecar>]<BaseName><SizeLabel><FineTune><Version><Encoding><Type><Shard>.gguf
Its own examples show how the parts read. Grok-100B-v1.0-Q4_0-00003-of-00009.gguf is base name Grok, 100B parameters, version 1.0, Q4_0 encoding, shard 3 of 9. mmproj-Qwen2-VL-7B-v1.0-F16.gguf is a multimodal projector sidecar for a 7B Qwen2-VL model, stored in F16. The specification says the convention is “not intended to be perfectly parsable in the field”, and real repositories often skip parts of it.
| Filename component | What it helps identify | What it does not settle |
|---|---|---|
Sidecar prefix (mmproj, mtp) | A module loaded alongside a base model | Whether it matches your base model’s version |
| Base name | The model family in the file | Whether it is the original publisher’s upload |
| Size label | The advertised parameter-size variant | Total memory needed at your chosen context length |
| Fine-tune or version | Which derivative or revision was converted | Its license or suitability for your task |
| Encoding label | How most stored weights are represented | A universal accuracy or speed ranking |
Type (LoRA, vocab) | An adapter or a vocabulary-only file | Whether that file works on its own (it does not) |
| Shard suffix | One part of a split download | Whether you have downloaded all required parts |
Start with the repository’s model card. Confirm the base model, conversion provenance, intended runtime and license. A conversion changes the storage format without granting additional rights to use the underlying model. Our open-weight versus open-source explanation covers the license distinction, and the open-model catalog lists published conditions model by model.
A real download list, worked through
The ggml-org/gemma-3-4b-it-GGUF repository shows what a short list looks like in practice. Its GGUF metadata reports 3,880,099,328 parameters. We divided each file’s size by that count to get the stored bits per weight.
| File | Size | Bits per weight | What it is |
|---|---|---|---|
gemma-3-4b-it-Q4_K_M.gguf | 2.49 GB (2.32 GiB) | 5.13 | Main model, 4-bit K-quant mix |
gemma-3-4b-it-Q8_0.gguf | 4.13 GB (3.85 GiB) | 8.52 | Main model, 8-bit |
gemma-3-4b-it-f16.gguf | 7.77 GB (7.23 GiB) | 16.02 | Main model, 16-bit |
mmproj-model-f16.gguf | 0.85 GB (0.79 GiB) | n/a | Image projector, needed only for image input |
Three lessons follow from those numbers.
- “Q4” does not mean 4 bits. The Q4_K_M file stores 5.13 bits per weight on average. Hugging Face’s documentation puts the Q4_K block format itself at 4.5 bits per weight, because every block of 32 weights also carries a scale and a minimum; the remainder comes from tensors the file stores in other formats and from metadata. Q8_0 lands at 8.52 because each 32-weight block also stores a scale. Our Q4_K_M vs Q5_K_M comparison looks at that choice in detail.
- You need one main file, not all three. The Q4_K_M, Q8_0 and f16 files are alternatives. Pick one.
- Image input adds a file. The projector’s name does not repeat “gemma-3-4b”, so it is easy to miss. For text and images with the Q4_K_M model, the download is 2.49 + 0.85 = 3.34 GB before any memory for the context cache.
Repositories also use prefixes the specification does not define. In ggml-org/Qwen3-8B-GGUF, files beginning dflash- and dspark- are converted from separate source models that the model card lists alongside Qwen3-8B, and ggml-org/gpt-oss-20b-GGUF carries eagle3- files next to the main model. None of these is the main model. When a prefix is not mmproj or mtp, the model card is the only reliable guide to what it does.
Quantization: a smaller file with a trade-off
Quantization stores weights at lower precision to reduce their footprint. A GGUF repository can offer several encodings of the same model, each a different compromise between storage, memory and retained numerical detail.
A lower nominal bit width does not tell you how much quality a particular task will lose. That depends on the model, the conversion and the workload. For a first download, use a variant the model publisher or your runtime’s documentation recommends. If you compare encodings later, keep the base model, prompt and generation settings fixed, and test the tasks you actually need. For background on the number formats themselves, see model precision formats.
File size is not the full memory requirement
A file can fit on your disk and still exceed the memory available to the application. Runtime allocations and the context cache also consume memory, and the cache grows with context length; our KV cache explainer shows how to estimate it.
The OSBBD memory calculator estimates weights plus a stated allowance and excludes the context cache. Use it to narrow a shortlist, then check the application’s memory guidance for the context length you intend to use.
Also distinguish system RAM from GPU memory. Which pool matters depends on the runtime and how much work it places on the GPU. Our guide to running AI locally covers those hardware limits.
Split files and companion files
A shard suffix identifies a piece of a split model. The specification requires five-digit numbers starting at 00001, so 00001-of-00009 is the first of nine parts, not one of nine runnable variants. Download the full set before loading.
Sidecars solve a different problem. The specification defines two: mmproj, a multimodal projector that turns image or audio input into something the language model can use, and mtp, multi-token-prediction heads used as a speculative-decoding draft module. It notes that MTP weights are often inside the base model, in which case there is no separate mtp- file. A sidecar must match the base model’s architecture and version; a similarly named file from another release is not a safe substitute.
The Type field marks two more non-standalone files: LoRA, an adapter that modifies a base model, and vocab, a file with only vocabulary data and metadata. Neither gives you a complete model.
A practical download checklist
- Identify the exact base model and revision in the model card, and read its license.
- Confirm your runtime supports that architecture and any sidecar you need.
- Choose one encoding that fits your memory, leaving room for the context cache.
- Check for shard suffixes and download every part.
- Add a projector only if you need image or audio input, and take the one the repository pairs with your model.





