Model precision formats describe how software stores or computes with a model’s numbers. Labels such as FP32, BF16, FP16, INT8 and 4-bit affect memory use, hardware compatibility and numerical behaviour, but they do not provide a universal quality ranking. For local AI, the useful question is which values are stored at lower precision, which precision the runtime uses for calculations, and whether the chosen backend supports that combination.

Precision is only one part of the memory budget. The context cache, temporary buffers and runtime overhead still matter, which is why a file that fits in RAM can fail when the requested context grows.

FP32 is the reference, not always the practical default

FP32 uses 32 bits for each floating-point value. It provides a broad numerical range and more precision than the smaller formats discussed here, but model weights stored in FP32 consume roughly four bytes per parameter before other memory costs are counted.

That footprint is often unnecessary for inference. A seven-billion-parameter model stored entirely as FP32 weights alone would require about 28 GB in decimal units. The estimate does not include the context cache or the runtime’s working allocations.

PyTorch still creates many floating-point tensors as FP32 by default, but its mixed-precision guidance says most modern neural-network work does not require every operation to use full FP32. The important caveat is numerical sensitivity: reductions, linear algebra and some training workloads can need more range or precision than a blanket conversion provides.

BF16 keeps range while reducing precision

BF16, or bfloat16, uses 16 bits per value. It retains the same exponent width as FP32, so its dynamic range is similar, while using fewer bits for the significand. In plain terms, BF16 can represent very large and very small magnitudes without the same overflow pressure as FP16, but each represented number is less precise.

PyTorch’s mixed-precision guidance recommends trying BF16 when FP16 overflows and notes that the formats can behave differently even on hardware that supports both. Hardware support is decisive. A processor or accelerator that lacks efficient BF16 operations may convert values or run more slowly than the label suggests.

For local inference, a model card that names BF16 often describes the published checkpoint or a recommended compute type. That does not mean a compact 4-bit download will occupy 16 bits per stored weight. Storage precision and compute precision can differ.

FP16 offers more precision but a narrower range

FP16 also uses 16 bits, with more significand bits and fewer exponent bits than BF16. That gives FP16 finer precision within its representable range but makes overflow and underflow more likely.

During training, PyTorch’s Automatic Mixed Precision pairs selected FP16 operations with FP32 operations and can use gradient scaling to prevent small gradients from flushing to zero. Inference does not need gradient scaling because there is no backward pass, but a runtime can still select different compute types for different operations.

Do not assume that an FP16 checkpoint will be exactly half the complete memory of an FP32 run. The weights may be close to that ratio, while the context cache, buffers and software allocations follow their own formats. The guide to KV cache memory explains why long contexts can dominate after a model is loaded.

INT8 reduces the weight footprint with calibration choices

INT8 quantization represents selected values with eight-bit integers plus scaling information that maps them back to useful numerical ranges. The straightforward weight estimate is about one byte per parameter, but scales, metadata and unquantized components add overhead.

Hugging Face’s bitsandbytes documentation describes an eight-bit method that keeps outlier features in a higher-precision path. It can roughly halve the memory used by 16-bit weights, but the exact result depends on the model and implementation. The documentation also warns that eight-bit training supports training extra parameters rather than updating all quantized weights directly.

Quantization is an algorithm, not a file extension. Two files both called INT8 can use different grouping, scale choices or outlier handling. Compare the runtime, conversion method and model revision instead of treating the bit count as the whole specification.

Four-bit weights trade more numerical detail for memory

Four-bit formats can bring weight storage close to half a byte per parameter before scales and other overhead. They make larger models practical on consumer hardware, but they discard more information than eight-bit or 16-bit storage.

The bitsandbytes integration supports two commonly discussed four-bit types, FP4 and NF4. Its documentation recommends NF4 for training a four-bit base model because NF4 is designed for weights drawn from a normal distribution. That recommendation belongs to a particular method and use case. It does not establish that every NF4 conversion will outperform every other four-bit format in local inference.

QLoRA-style fine-tuning loads a frozen base model in four-bit form and trains added low-rank adapter parameters. It is not the same as updating every original weight in four-bit arithmetic. If you are downloading a local model for inference, the repository should state whether the file contains a complete quantized base model, an adapter or both.

Our GGUF file guide explains how container labels, shards and companion files fit around this decision. A GGUF quantization code is more specific than a bare bit count, but it still needs a runtime that supports the model architecture and encoding.

Storage type and compute type can differ

A runtime can store weights in four or eight bits, then dequantize blocks into BF16 or FP16 while performing matrix operations. Activations and the KV cache may use still other types. As a result, a settings panel can show both a quantization format and a compute dtype without contradiction.

This is also why file size is an incomplete performance prediction. A heavily quantized file reduces memory traffic, yet dequantization work and hardware support can change speed. A larger format may run faster when the accelerator has efficient native operations for it.

Check four separate values when documentation exposes them:

  • the format used to store the model weights;
  • the type used for matrix calculations;
  • the type used for activations and the KV cache;
  • the placement of those values across GPU memory and system RAM.

The local AI hardware guide covers the difference between VRAM and system memory. The open-model memory calculator can estimate weight storage plus a stated allowance, but its result excludes the context cache.

Choose the smallest format that still works for your task

Start with the model publisher’s supported formats and the runtime’s hardware table. Eliminate downloads that your software cannot load or that exceed available memory once context and overhead are included.

Then compare candidate formats on the work you actually need. Keep the base model, prompt, context and generation settings fixed. Check instruction following, factual extraction, structured output and any languages central to the job. A one-number benchmark cannot tell you whether a conversion preserves the behaviour your workflow depends on.

If the result fails, the cause may be model capability rather than precision. Test the same model at a higher-precision encoding before moving to a larger model. Conversely, if a 4-bit version performs adequately, a 16-bit copy may consume more memory without improving the result enough to matter.

Avoid claims based only on filenames. Read the model card, conversion notes and runtime documentation. A label such as 4-bit leaves open which values were quantized, how scales are stored and which compute type will be used.