ThinkBit shrinks Qwen3-14B to 3.74 GB and still scores 94.8 on MATH-500

Two-bit LLM compression has long been able to look good on paper: perplexity barely rises, and multiple-choice benchmarks pass. Yet such models often lose the ability to reason. The ThinkBit team, whose paper is linked to the Microsoft organization on Hugging Face, attacked the problem from the other end and designed its method for long reasoning chains. On Qwen3-14B the result is this: a 3.74 GB file scores 94.8 on MATH-500, while the original 29.5 GB bfloat16 model scores 97.5. The size difference is almost 8x; the quality difference is under three points.

Why reasoning breaks first

Reasoning models emit thousands of tokens per answer. Generation speed here is limited not by compute but by memory bandwidth: every token requires reading all the weights. Storing them in two bits instead of sixteen can potentially cut inference cost by 8x. But the standard metrics for evaluating 2-bit methods, perplexity and multiple-choice QA, say little about whether a model will survive a hard multi-step task. A small error in the weights accumulates over a long chain of tokens, and the final answer drifts off course.

ThinkBit stays strictly scalar. Each weight is represented as an integer codeword multiplied by a bfloat16 scale shared across a group of weights. There is no vector quantization with large codebooks, which makes the format simpler to implement in runtimes.

Four stages and a final arbiter

The pipeline is sequential, and each stage has its own role:

• Bit allocation. A sensitivity metric assigns each layer to one of three tiers: ternary, 2-bit or 3-bit. Critical layers get more bits; robust ones get fewer.
• Codeword search. A layer-by-layer algorithm based on Gumbel-softmax searches for the optimal codewords block by block.
• Scale recalibration. A layer-wise scheme cannot see how quantized layers affect one another. So the codewords are frozen, and the scales are tuned against a loss based on knowledge distillation from a teacher model.
• Short QAT. Quantization-aware training lets the model reproduce the base model's distribution by reassigning codewords.
• Checkpoint selection. A paired reasoning test on training problems decides which result to keep: the recalibration output or the QAT output. The choice is made on reasoning ability, not on perplexity.

The last step is the most unusual. QAT does not always improve reasoning, and the authors check this directly rather than assume it.

Numbers on Qwen3-1.7B and Qwen3-14B

On Qwen3-1.7B, at a comparable size, the ThinkBit recalibration checkpoint scores 46.7 on MATH-500. EfficientQAT on the same data gets 30.0. After QAT on reasoning traces, the result rises to 73.0. The file takes up 0.52 GB. It is smaller than any published 2-bit GGUF of this model and more accurate on MATH-500.

The full list of reported metrics:

• Qwen3-1.7B, ThinkBit recalibration: 46.7 on MATH-500
• Qwen3-1.7B, EfficientQAT on the same data: 30.0
• Qwen3-1.7B, ThinkBit after QAT on reasoning traces: 73.0
• Qwen3-1.7B size in ThinkBit: 0.52 GB, smaller than all published 2-bit GGUFs
• Qwen3-14B, ThinkBit: 3.74 GB, 94.8 on MATH-500
• Qwen3-14B, bfloat16: 29.5 GB, 97.5 on MATH-500

Actual compression of the 14B model is roughly 7.9x, close to the theoretical ceiling of 8x for going from 16 bits to 2. The method scales from 1.7B to 14B without losing its overall logic.

What this means for people who run models themselves

A strong reasoning model at 3.74 GB fits on an 8 GB GPU with room left for the KV-cache. Previously, this class of tasks required a 24–32 GB card. For local use and for services with long generation, this is a direct saving: less memory per replica, more replicas on the same card, higher throughput.

When evaluating your own quantized models, it is worth adding reasoning tasks such as MATH-500 to the test suite and comparing more than just perplexity. The 1.7B results show how easily metrics diverge: perplexity would most likely not have revealed the gap between 30.0 and 46.7 at a similar size. For now, no models, datasets or Spaces are linked to the paper, so reproducing the results means working from the method description.