4-Bit Weight Quantization: Why the Memory System Decides

If you have tried to run a language model on a Jetson module or a similar edge board, you have seen the two limits quickly. The model does not fit, and even when it fits, token generation is slow. Both limits are properties of the memory subsystem, not of the model. 4-bit weight quantization is the technique that addresses both, and it is best understood as a data-movement optimisation: it reduces the number of bytes that must cross the memory bus to produce one token. This article explains what 4-bit weight quantization does to the data path, why the obvious implementation damages the model, and why the achievable speed on a real product is decided by the memory system rather than by the quantiser.

The bottleneck is DRAM bandwidth, not arithmetic

Generating tokens is not a compute-heavy operation. The model processes one token at a time, so each weight matrix is read from memory and used for a single small matrix-vector product. Each weight is loaded and used roughly once. The arithmetic intensity of the generation stage in FP16 is approximately 1 operation per byte loaded.

That single number determines everything else. Modern accelerators offer far more arithmetic throughput than memory bandwidth, so at an intensity of 1 the compute units sit idle waiting for weights. Making the arithmetic faster changes nothing. Reducing the bytes that move from DRAM to the compute units changes the runtime almost proportionally. This is an ordinary memory-hierarchy result, and it is the reason quantization, rather than faster math, is the main optimisation for edge inference.

What 4-bit weight quantization does to the data path

The approach is weight-only quantization. Weights are stored as 4-bit integers; activations stay in 16-bit floating point. The weights are dequantised back to FP16 inside the kernel, immediately before use. This is written as W4A16. Only the weights are compressed because only the weights are the bandwidth problem.

Quantization is done per group rather than per tensor. A group size of 128 is the common choice: every 128 weights along a channel share one scale and one zero point. Small groups keep the error low, and the scale metadata stays small relative to the weights it covers.

The gain is bytes. Sixteen bits become four, so the weight footprint drops about four times. Llama-2-70B needs roughly 140 GB in FP16 and about 35 GB at 4 bits. The first number does not fit on any edge module; the second does.

On an edge SoC, memory is a shared and contended resource

This is where edge deployment stops resembling a desktop benchmark. A Jetson AGX Orin 64 GB module carries 64 GB of 256-bit LPDDR5 with 204.8 GB/s of bandwidth, and it is unified memory: there is no discrete VRAM. The CPU, the GPU, the ISP, the display engine and every DMA master read from the same physical SoC DRAM through the same controller. CPU allocations directly reduce the memory available to the GPU.

Two consequences follow, and neither appears in a model benchmark.

First, bandwidth sets a hard ceiling you can calculate before writing any code. If every generated token must read all 35 GB of a 4-bit 70B model, then 204.8 GB/s divides to about 5.8 tokens per second as an upper bound. That figure ignores KV cache traffic and assumes nothing else touches memory, so the real number is lower. It is still the right first calculation to run, because no kernel optimisation will beat it.

Second, that bandwidth is contended. On a real product the model is not alone: camera streams, the ISP, the display pipeline and the page cache are all moving bytes through the same DRAM at the same time. A multi-camera pipeline consumes bandwidth that the inference kernel then does not get. This is why a token rate measured on an idle board does not survive integration, and why the fix is usually a memory-system decision rather than a model change.

Loading is its own problem. Bringing tens of gigabytes of weights into memory involves the page cache, the reclaim path, and first-token latency that has nothing to do with the model architecture. On a module with a fixed memory budget shared by every subsystem, whether the weights stay resident is a memory-management question. Reading and reasoning about the memory subsystem at this level is what our Linux Kernel Infrastructure training covers.

Why plain rounding damages the model

The obvious implementation is round-to-nearest: take each group, find its range, and map the values onto the available levels, 16 of them at 4 bits. At 4 bits this works acceptably. At 3 bits it fails. Quantising OPT-6.7B to 3 bits with round-to-nearest and a group size of 128 pushes WikiText perplexity to 43.2. The model is close to unusable.

The reason is that weights are not equally important. A small fraction of the weight channels matter far more than the rest, and rounding treats them identically. If you keep just 1% of the weights in FP16 and quantise the remaining 99% to 3 bits, perplexity recovers from 43.2 to 13.0. One percent of the weights accounts for almost the entire loss.

Finding important weights by activation, not by weight

The useful part is how you decide which 1% is important. The intuitive answer is to look for large weights. That answer is wrong. The channels that matter are those carrying large activation magnitudes, because a weight’s quantization error is amplified by the activation it multiplies. A modest weight consistently multiplied by a large activation contributes more error than a large weight multiplied by near-zero input. The selection is therefore made by running a small calibration set through the model and measuring the average activation magnitude per input channel.

Scaling instead of mixed precision

Keeping 1% of weights in FP16 recovers accuracy but produces a mixed-precision tensor, and that is a memory-layout problem. A tensor that is mostly INT4 with scattered FP16 values cannot be packed into a regular layout, forces irregular access, and breaks the contiguous, coalesced reads the kernel depends on. The format must stay uniform.

The alternative is to scale. Multiply the salient input channels by a factor s greater than 1 before quantising, and divide the corresponding activations by s to keep the mathematics equivalent. Scaling a channel up before rounding reduces its relative quantization error, because the group’s step size barely changes while the channel’s values grow. The salient channels are protected and every weight remains a plain 4-bit integer. On the OPT-6.7B example above, this reaches the same perplexity of 13.0 as the mixed-precision version, with a layout the hardware can actually stream.

The factor is searched, not guessed, and it does not increase without limit: enlarging the salient channels widens the group’s range, increases the step size, and amplifies the error of the non-salient channels sharing that group. In the reported ablation the best result appears at s = 2.

The speedup is won in the kernel, not the quantiser

A four times smaller weight tensor does not deliver a four times speedup by itself. The dequantisation has to happen somewhere, and a naive implementation writes dequantised FP16 weights back to DRAM before the matrix multiply, which reloads them and returns the traffic to exactly what quantization removed. The dequantisation must be fused into the matrix-multiply kernel so weights are unpacked in registers and consumed immediately. The compression only counts if the FP16 values never reach memory.

Two further details are pure systems work. The small kernels involved run in about 0.01 ms on a modern GPU, comparable to the kernel launch overhead itself, so fusing operations to reduce launches produces a direct gain. And a byte-addressed machine has no native 4-bit load: the values must be packed in the order the unpacking instructions consume them, which is a data-layout and instruction-set decision specific to the target.

What the numbers look like

Accuracy at 4 bits with a group size of 128, as WikiText perplexity where lower is better:

• Llama-2-7B: 5.47 in FP16, 5.60 with activation-aware quantization, against 5.73 for round-to-nearest.
• Llama-2-70B: 3.32 in FP16, 3.41 quantised.
• Mistral-7B: 4.14 in FP16, 4.30 quantised. Mixtral-8x7B: 5.94 in FP16, 6.05 quantised.

At 3 bits the gap between methods widens: Llama-2-7B reaches 6.24 with activation-aware quantization against 6.66 for round-to-nearest. Accuracy is roughly preserved at 4 bits regardless of method, so the quantiser’s quality matters mainly when the bit budget is tighter.

On the performance side, with fused 4-bit kernels the measured average speedup is 3.2 to 3.3 times over the Hugging Face FP16 implementation, consistently across desktop, laptop and mobile GPUs. A 13B model runs at 30 tokens per second on a laptop RTX 4070 with 8 GB of memory. Llama-2-70B runs on a single Jetson Orin with 64 GB.

Where it breaks down

• The speedup depends on batch size 1, where the workload is memory-bound. In batched serving the arithmetic intensity rises and the analysis changes; W8A8 quantization, which also compresses activations and uses integer arithmetic, becomes more relevant.
• The accuracy figures depend on the group size. A group size of 128 carries per-group scale metadata, so the true average is above 4 bits per weight. Larger groups save metadata and lose accuracy.
• Three-bit quantization is usable but not free. Llama-2-7B still moves from 5.47 to 6.24.
• A calibration set is still required to measure activation magnitudes.
• The reported speedups are GPU results measured on an otherwise idle device. An NPU or DSP with its own quantised data path behaves differently, and the packing decisions do not transfer.

What this means for embedded and kernel engineers

Model deployment on edge Linux is largely a memory-system problem wearing a machine-learning label. The decisions that determine whether a model runs at an acceptable rate on your board are the bit width, the group size, whether the dequantisation is fused, how the weights are packed for the target instruction set, and how much DRAM bandwidth the rest of the product has already committed. None of these are model-training decisions, and all of them are the kind of reasoning an engineer applies to any DMA-heavy pipeline.

This also explains why the tooling looks the way it does. 4-bit weight quantization with activation-aware scaling is supported in Hugging Face Transformers, NVIDIA TensorRT-LLM, vLLM, Intel Neural Compressor and other stacks, so in most cases you are selecting and configuring it rather than implementing it. The value is in knowing which parameter affects which cost: bit width and group size determine bytes, kernel fusion and packing determine whether those bytes turn into latency, and the memory system you share with the camera and display determines what is left.

Key takeaways

• Token generation is memory-bandwidth-bound, with arithmetic intensity around 1, so bytes moved decide the token rate.
• 4-bit weight quantization with a group size of 128 cuts the weight footprint about four times and keeps accuracy close to FP16.
• On a unified-memory SoC such as Jetson Orin, DRAM bandwidth is shared with the CPU, ISP and display, so the benchmark token rate is an upper bound, not a product figure.
• Roughly 1% of weight channels cause most of the quantization error; they are identified by activation magnitude, not weight magnitude.
• The memory saving becomes a speedup only when dequantisation is fused into the matrix-multiply kernel and the weights are packed for the target instruction set.

Frequently asked questions

Why is 4-bit weight quantization treated as a memory-system problem rather than a model problem?
During single-token generation the arithmetic intensity is about 1 operation per byte loaded, so the compute units wait on DRAM. The number of bytes read to produce a token, not the arithmetic, sets the token rate. Compressing the weights reduces those bytes directly.

Why are the important weights chosen by activation magnitude rather than weight magnitude?
A weight’s quantization error is multiplied by the activation it meets. A moderate weight that consistently receives a large activation contributes more error than a large weight that receives near-zero input, so activation statistics identify the salient channels more accurately.

How much accuracy is lost at 4 bits?
Little. Llama-2-7B moves from 5.47 to 5.60 WikiText perplexity, and Llama-2-70B from 3.32 to 3.41, at a group size of 128. At 3 bits the loss is larger and the choice of quantization method matters much more.

Why is the token rate on a real product lower than the published benchmark?
Benchmarks run on an otherwise idle device. On a unified-memory SoC the model shares one DRAM controller with the CPU, camera, ISP and display, so bandwidth those subsystems consume is bandwidth the inference kernel does not get.

Further reading

• Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, Song Han. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. MLSys 2024 (Best Paper Award). The accuracy and speedup figures in this article are from this work.
• MLSys 2024 proceedings entry for the same work.
• MIT Han Lab project page, including the TinyChat inference framework and the Jetson Orin measurements.
• NVIDIA Jetson AGX Orin Series Technical Brief — memory architecture and bandwidth figures.
• llm-awq reference implementation on GitHub.