How does model quantization work?

Keep the network, change the representation

A model contains numerical parameters. Quantization changes how selected values are represented and used; it does not inherently remove layers, reduce the number of parameters, or create a different model architecture.

Imagine marking a measured length on a fine ruler and then replacing it with a ruler that has fewer marks. Nearby values now share a representable value. A scale tells the system how those marks relate to the original units.

For a simple uniform quantizer, the sequence is choose a range → map to a code → round and bound the code → reconstruct a value. Real methods can use groups, different precision for different quantities, and choices optimized to preserve useful model behaviour.

Move eight weights onto a coarser ruler

The experiment uses eight authored weights and a fixed range of −1 to 1. Two bits supply four code values, 0 through 3; four bits supply sixteen. Each code maps back to one level across the chosen range.

Change the bit width and inspect original weights, stored codes, reconstructed values, and one dot product against a fixed input. Then introduce a weight of 1.8 without changing the range. Predict whether additional bits can restore it.

More levels can reduce rounding error within the range. They cannot recover the portion of a value clipped outside that range: 1.8 still reconstructs as 1. The range and scaling policy are as important as the label “4-bit.”

The displayed errors and dot products are computed locally. They are not LLM accuracy, perplexity, latency, or an implementation of GPTQ, AWQ, or LLM.int8.

Compression needs its decoding information

Eight four-bit codes occupy a four-byte weight payload if packed. Reconstructing the values also needs the range or scale and offset. Actual artifacts have metadata, alignment, grouping, and sometimes values retained at higher precision.

The arithmetic is straightforward at larger scale: seven billion values at four bits each occupy 7,000,000,000 × 4 / 8 = 3.5 billion bytes of raw codes. That is 3.5 decimal GB before overhead, not a claim that a complete seven-billion-parameter model runs in 3.5 GB.

Serving also holds activations, attention/KV state, temporary buffers, runtime allocations, and concurrency-dependent data. Compressing weights does not eliminate those costs. A long-context workload can still run out of memory.

Real methods protect different parts of the calculation

GPTQ uses approximate second-order information to select weight quantization while accounting for reconstruction error. It is more deliberate than independently rounding every weight to the nearest value.

AWQ uses activation information to guide weight scaling and protect important behaviour. The size of a weight alone is not enough to explain its influence on outputs.

LLM.int8 combines an eight-bit path with higher-precision handling for important outlier features. A deployment described as low-bit can therefore contain several numerical precisions.

These methods illustrate why bit width alone is an incomplete compatibility or quality description. Distinguish what is quantized—weights, activations, or cache—from the calibration procedure, grouping, reconstruction rule, accumulation precision, and supported execution kernels.

Smaller storage is not a universal speedup

A runtime must execute the representation efficiently. Packing, unpacking, conversion, memory transfer, and available hardware kernels influence latency and throughput. A compact artifact can save memory without improving every workload's elapsed time.

Likewise, a small numerical error is not a product-quality guarantee. Repeated transformations can amplify differences; a close pair of output scores can change their ordering. Compare the original and quantized artifacts on the actual tasks, including important languages, long inputs, schemas, tools, and severe failure cases.

Keep model identity and test conditions explicit. Quantization is an engineering trade-off to measure, not a permanent claim that a smaller file is equivalent to its source in every setting.

The uniform quantizer used in this example

For bit width b, let levels = 2ᵇ, range minimum −1, and range maximum 1. The level spacing is scale = 2 / (levels − 1).

The code is round((weight + 1) / scale), bounded to 0 … levels − 1. Reconstruction is −1 + code × scale. The toy uses ordinary JavaScript numbers for calculation and estimates packed payload size; it does not store a real low-bit checkpoint.

The dot product multiplies each weight by its corresponding fixed input and sums the products. Comparing the original and reconstructed results exposes one numerical consequence. Different inputs or a different calibration range can change that consequence.