What is quantization in deep learning?

What is quantization?

Quantization is the process of storing a neural network's weights, and sometimes its activations, using fewer bits than the 16 or 32-bit floating-point numbers they were trained in. A weight stored in 4-bit integer format takes a quarter of the memory of its 16-bit original. Large language models are mostly a long list of weights moved from memory into a processor. Cutting the bits per weight cuts both the memory footprint and, often, how long inference takes.

The catch is precision. Squeeze a 16-bit number into 4 bits and some information is lost. That loss can show up as higher perplexity, a measure of how well a model predicts text, or as lower scores on tasks. Most of the work in this field since 2022 has been about losing as little of that accuracy as possible.

What is post-training quantization?

Post-training quantization, or PTQ, compresses an already-trained model without retraining it. The alternative, quantization-aware training, bakes the lower precision into the training process itself. That is expensive to run on a model with tens of billions of parameters. The GPTQ paper (Frantar et al., 2022) notes that PTQ methods typically need just a few thousand calibration data points and a few hours of compute. That is why PTQ has become the default for compressing large language models.

The simplest PTQ approach is round-to-nearest, which just rounds each weight to the closest value on the lower-bit grid. The GPTQ paper found that round-to-nearest works fine down to 8 bits but collapses at 3 bits. On OPT-175B it pushed perplexity on the WikiText2 benchmark from 8.34 in 16-bit to 7,300, versus 8.68 for GPTQ at the same bit width.

Why do large models quantize badly?

The LLM.int8() paper (Dettmers et al., 2022) traced the problem to outlier features: a handful of dimensions in a transformer's hidden states that carry values up to 20 times larger than the rest. Below 6.7 billion parameters these outliers are rare and scattered. At 6.7 billion parameters and beyond, the paper found a sudden phase shift. Outliers appear in all layers and about 75% of sequence positions, concentrated in as few as six feature dimensions per model.

Those outliers matter disproportionately. The same paper found that zeroing them out dropped the top-1 attention probability by more than 20% and raised perplexity by 600 to 1,000%. Zeroing an equal number of random features, by contrast, changed perplexity by just 0.1%. A quantization scheme that treats all weights the same way will blow its precision budget on values that barely matter, while mangling the few that do.

GPTQ vs AWQ: two ways to quantize

GPTQ (Frantar et al., 2022) quantizes one layer at a time, using second-order information, roughly, how sensitive the output is to each weight, to decide how to round each weight. It then adjusts the remaining unquantized weights in that layer to compensate for the error. It quantized OPT-175B and BLOOM-176B to 3 or 4 bits in about four GPU hours each. At 4-bit on OPT-175B's WikiText2 score, it reported losing only 0.03 perplexity points, against a 2.2-point loss for round-to-nearest.

AWQ (Lin et al., 2023) takes a different view. It finds that protecting roughly 1% of weights, the ones that correspond to large activation values, preserves most of the accuracy, and it is activation magnitude, not weight magnitude, that identifies them. Rather than keep those weights in higher precision, which complicates hardware support, AWQ scales up the salient weight channels before quantizing. The paper's analysis shows this lowers their relative rounding error. On Llama and Llama-2 models from 7 billion to 70 billion parameters, AWQ's INT4 perplexity beat both round-to-nearest and GPTQ.

AWQ also skips GPTQ's layer-by-layer reconstruction step. The paper reports that this lets it generalize better to instruction-tuned and multi-modal models. It attributes GPTQ's weaker showing there to its reconstruction process overfitting the calibration set.

What is GGUF?

GGUF is a file format used by the llama.cpp project to package quantized model weights for local inference. The AWQ paper references it as one of the systems performing group-wise INT4 quantization to cut memory traffic. None of the documents behind this page describe GGUF's internal design in detail, so specifics of its format are outside this page.

A study by Wei-Liang Liu and Cheng-Yen Tsai checked the 200 most-downloaded Hugging Face GGUF repositories, 198 of which were in scope, and found that none recorded the build environment needed to independently re-derive the quantized files. On four repositories that did disclose their inputs, the model card alone could not reproduce any Q4_K_M file; adding two undisclosed settings, the floating-point-contraction option and, for hybrid models, the per-tensor quantization recipe, re-derived seven of eight target files bit-for-bit on x86 and five of eight on arm64. The calibration imatrix, a file used in some quantization recipes, had to be shipped and hash-bound, since recomputing it did not match the original. Model signing authenticates a model's full-precision weights, but the study argues that guarantee does not extend to the quantized GGUF files most users actually run.

What accuracy is lost, and how is it measured?

All three papers measure quantization damage mainly through perplexity, a score for how well a model predicts held-out text, because it is sensitive to even small degradation. On OPT-175B, the LLM.int8() paper found that its 8-bit method kept accuracy identical to the 16-bit baseline across WinoGrande, HellaSwag, PIQA and LAMBADA. Prior 8-bit methods, by contrast, degenerated into random performance at that scale.

Going lower costs more. GPTQ reported maintaining near-baseline perplexity at 4-bit across OPT and BLOOM, losing roughly 0.3 to 0.6 points at 3-bit on OPT-175B and BLOOM-176B. It needed grouped quantization, applying separate scales to small blocks of weights, to make 2-bit quantization usable at all. AWQ's own comparisons on Llama-2 models showed its INT3 and INT4 perplexities consistently closer to FP16 than GPTQ's at matching bit widths and group sizes.

Where is quantization used today?

The AWQ paper reports adoption across Hugging Face Transformers, NVIDIA TensorRT-LLM, Microsoft DirectML, Google Vertex AI, Intel Neural Compressor, Amazon SageMaker, AMD, FastChat, vLLM and LMDeploy. It says AWQ enables Falcon-180B to run on a single H200 GPU. Its companion inference system, TinyChat, reports over 3x speedup against Hugging Face's FP16 implementation, and running a 70-billion-parameter Llama-2 model on a mobile GPU.

GPTQ's paper reports fitting a 3-bit OPT-175B model, plus its attention cache, into a single 80GB A100 GPU, versus five such GPUs for FP16. It reports an end-to-end latency reduction of about 3.25x on an A100 and 4.5x on a more affordable A6000 GPU. For engineers, the question is rarely whether to quantize, but which scheme and bit width a given GPU budget and accuracy bar can tolerate.