Quantization

Quantization makes an AI model smaller by storing its numbers with less precision, like rounding prices to whole euros. The model needs less memory and often runs faster, so it fits on a cheaper computer or even a laptop. That helps a business that wants to run a model on its own hardware, for example to keep customer data in-house. The price is some loss of quality.

How does it work?

A model consists of billions of numbers, its weights. They are stored as 32-bit floating-point numbers or, increasingly, as 16-bit ones (fp16 or bf16), and quantization stores them in fewer bits, down to integers such as int8 or int4 (Hugging Face). An 8-bit integer can take 256 different values and a 4-bit integer 16, against 65,536 for 16 bits. Each weight is rounded to the nearest available value. The term comes from signal processing, where quantization means mapping a large set of values onto a smaller one, with rounding as the typical example.

To map the integers back, the method keeps a scale and a zero point per tensor or per channel. In the affine scheme of the Hugging Face concept guide, a value is x = S * (x_q - Z), with x_q the stored integer, S the scale and Z the zero point. The saving follows from the bits: the weights of a model with 70 billion parameters take about 140 GB at 16 bits and about 35 GB at 4 bits.

When is a model quantized?

• After training (post-training quantization, PTQ). The finished model is converted without further training, sometimes with a small calibration set to measure the ranges of values. GPTQ (Frantar et al., 2022) calibrates on 128 text fragments and brought OPT-175B and BLOOM-176B to 3 or 4 bits in about four GPU hours, with negligible loss of accuracy. AWQ (Lin et al., 2023) found that protecting 1% of the weights, picked by the activations they meet, greatly reduces the rounding error.
• During training (quantization-aware training, QAT). Training simulates the rounding, so the model learns to cope with it. Google trained Gemma 3 for about 5,000 steps this way, which cut the loss in quality at 4 bits, measured as perplexity, by 54% (Google, April 2025).

Large models need extra care. LLM.int8() (Dettmers et al., 2022) found that from about 6.7 billion parameters, a few feature dimensions take extreme values, and ordinary 8-bit methods fail. It keeps those dimensions in 16 bits and the other 99.9% in 8 bits, which halves the memory for inference without loss up to 175 billion parameters.

Does it make a model faster?

Often, because generating text is mostly limited by how fast the weights can be read from memory. GPTQ's kernels unpack the 3-bit weights while computing, and OPT-175B then generated a token about 3.25 times faster on one A100 than the 16-bit model on five, and 4.5 times faster on two NVIDIA A6000 cards than on eight. LLM.int8() saves memory but runs at about the speed of 16 bits, slightly slower end to end, because converting the values costs time.

Where do you meet it?

llama.cpp runs models locally with 1.5-bit to 8-bit integer quantization. Its models come as GGUF files, a binary format for fast loading from the maker of llama.cpp (Hugging Face Hub). For fine-tuning, QLoRA trains LoRA adapters on top of a 4-bit model, enough to fine-tune a model of 65 billion parameters on one 48 GB GPU (Dettmers et al., 2023). The KV cache, the memory a model uses while writing, can be quantized too.

Example

Suppose an accountancy firm wants a chat assistant on one office PC with an NVIDIA RTX 3090 (24 GB), so that client files never leave the building. Gemma 3 27B needs 54 GB for its weights at 16 bits and does not fit. Google's int4 version needs 14.1 GB, which leaves about 10 GB for the KV cache; Google points out that the cache comes on top of the weights.