rwkv-quant: keeping one matrix in bf16 removes 39% of RWKV-7 2.9B's quantization loss

Quantization of the RWKV model family, with its linear recurrent structure, remains less studied than quantization of transformers. The author of rwkv-quant, a tool that compresses RWKV-7 with two presets (about 2.1x and about 3.1x), noticed an anomaly. The aggressive COMPRESSION preset gives +4.04% perplexity on 2.9B but only +2.85% on 1.5B. Digging into the cause showed that a large share of the loss sits in a single 2560×2560 matrix. Storing it in bf16 costs 0.5% of the file and removes 39% of KL.

Two presets and six sizes

The study covers the RWKV-7 line in six sizes. Three pairs help separate the effect of width from the effect of depth: 0.4B/1.5B (same depth, twice the width), 2.9B/7.2B (depth 32, width 2560 versus 4096) and 7.2B/13.3B (width 4096, depth 32 versus 61).

• 0.1B: 12 layers, width 768
• 0.4B: 24 layers, width 1024
• 1.5B: 24 layers, width 2048
• 2.9B: 32 layers, width 2560
• 7.2B: 32 layers, width 4096
• 13.3B: 61 layers, width 4096

The REDUCTION preset is nearly lossless: projections in 8 bits with a symmetric layout and a group of 16 weights, cmix (FFN) in 6 bits, embedding and head in 8 bits. COMPRESSION is more aggressive: projections in 4 bits with an asymmetric superblock layout and a group of 32, cmix in 4 bits (ffn.key in 5), head in 5 bits, embedding in 6 bits. LoRA branches go in 6 bits in both presets (g in 8). In both, the scale and shift of each group are found by a search weighted by the mean square of the channel's input activation, E[x²].

Where 2.9B breaks

A leave-one-out breakdown of KL by matrix group showed that on 2.9B, o_proj accounts for 46% of all KL and the embedding for 17%. On 1.5B those shares are 16% and 3%, and the main contributor is cmix.value (37%). Within o_proj, 85–86% of the damage comes from a single matrix, the output projection of block zero.

A direct experiment on 2.9B (real path, MLX) confirmed this:

• baseline preset: KL 0.052048, +4.04% ppl;
• o_proj of layer 0 in bf16: KL 0.031966, +2.43% ppl, file larger by 9.4 MB;
• o_proj of layer 31 in bf16: no effect (+4.07%);
• all of o_proj in 5 bits: +3.54%, +26.2 MB;
• all of o_proj in 6 bits: +2.89%, +52.4 MB.

One matrix in bf16 works better than moving all of o_proj to 6 bits, and costs more than five times less. On other sizes the effect is weaker: KL drops by 7.6% on 0.1B, 1.5% on 0.4B, 0.4% on 1.5B and 3.9% on 7.2B. Only 0.1B and 2.9B are significant.

The input of this matrix carries a "massive channel". On 2.9B, eight channels hold 98% of the energy, and channel number 517 is the same across two series of checkpoints; on 1.5B it is channel 1202. So the channel is determined by width and architecture, not by a particular training run. But the channel's strength does not predict the damage: 7.2B has the highest concentration (99%) and the lowest baseline KL (0.023 versus 0.078 for 2.9B). Without AW weighting, 4 bits in o_proj wreck 2.9B: KL 0.340, +37.2% ppl.

The author shows that three natural local error metrics do not predict the damage. What worked was the error in the residual stream multiplied by the measured gain.

Calibration leakage and honest measurements

The first evaluation set (38 windows of 512 tokens, en/ru/sr) lay entirely inside the calibration corpus: 100% of 16-grams were shared. The scale of the leakage was measured on GPTQ. On 1.5B, the KL gain was −61.8% on the old set and −27.1% on the clean one, so the leakage inflated the result by a factor of 2.3. Automatic bit-width selection did not benefit from it: −16.1% versus −17.6%.

For that reason, sections 7–10 of the paper use only clean sets: Wikipedia en/ru/sr (36 windows), a closed Python/Swift code set (24 windows) and an open one (48 windows). Intervals are built with a paired bootstrap over windows, 20 000 resamples; the half-width of the 95% interval for KL is 7–12% of the value.

On top of this, the paper describes automatic bit-width selection using a "damage per byte" rule, which works in both directions, and GPTQ on the preset's grid without changing the format, kernels or file size. These methods combine almost multiplicatively. A separate section compares the results with GGUF and MLX at equal file size.

The practical takeaway is simple. If you quantize RWKV-7 for local use, especially on a Mac via MLX, keep the first layer's o_proj in bf16: that is 6–10 MB on 1.5B–2.9B models. Keep the embedding at 6 bits at minimum. Before trusting the numbers from a calibrated method, check that the calibration and evaluation texts have zero 16-gram overlap. And measure the contribution of individual matrix groups with leave-one-out rather than assuming that losses on a large model spread evenly.