Dharma-AI trimmed Qwen3-4B's vocabulary and cut inference cost by 77%

Brazilian team Dharma-AI has shown how much can be saved by touching something usually treated as a fixed part of a model: the tokenizer. In a production system that grades Portuguese essays on top of Qwen3-4B, a new vocabulary cut the average response from 350 to 101 tokens. Measured inference cost fell by 77.0%, and the benchmark score rose from 0.703 to 0.810. The base model, data and training recipe stayed the same. The method is called VocabX. The full paper has not been published yet, and the figures come from the current version of the preprint, which is under peer review.

Where the grading model was overpaying

The grader receives an essay, a topic and supporting texts, and returns JSON with a score and a comment for each competency. The tokenizer was still the original one from Qwen3-4B, meaning it was built for many languages and many types of text. It split into dozens of pieces exactly what the model writes in every response: the same JSON wrapper, the competency names, the standard phrasing of scores. On top of that, at every generation step the model computed probabilities over 151 669 vocabulary entries, most of which almost never appear in this task.

The cost here is asymmetric. During prefill, the model processes the entire input in one pass, and the bottleneck is compute. During decode, every token requires a full pass, the GPU mostly waits for data from memory, and reading the KV-cache adds to reading the weights. In the measured case, a generated token costs 7.4 times more than a token that is read. This ratio depends on hardware, batch size and context length, and can be higher. In a model the size of Qwen3-4B, the LM head matrix takes up almost 10% of the parameters, and it is read in full for every generated token. An unneeded vocabulary row costs as much as a frequent one.

Tokens chosen by price, not by frequency

Existing methods such as fast vocabulary transfer (FVT), multi-word tokenization (MWT) and Liquid AI's in-place BPE choose candidates by frequency and work on a single level. VocabX changes the criterion: each token saved is multiplied by the price of the field where it appears (system prompt, user message or model response). In this case, a token saved in the response weighs 7.4 times more than one saved in the input. The method works on two layers at once: sub-word and multi-word, where repeated fragments turn into a single macro-token. Five miners are used to find such fragments:

• long words;
• whole lines;
• expressions;
• full sentences;
• structures with punctuation, such as JSON and tags.

The last miner contributed the most: on average across eight corpora, and on code alone it added 17.5 points. Without all five, token reduction on the grader would have dropped by 55.6 points.

A separate problem is estimating the benefit from a sample of documents. An expression that appears 37 times in 50 000 documents lands in a sample of 400 documents in only 26% of cases. In a corpus of contracts, only 16.6% of candidates land in such a sample, and the rest get zero estimated benefit. The team computes exactly the part of the gain that can be computed over the whole corpus, and estimates the uncovered part statistically. On contracts, this raised token reduction from 44.7% to 52.8% with the same sample and the same budget.

The macro that swallowed the model's decision

For the measurements, the team trained seven versions of the grader on the same base, data, recipe and seed. Only the tokenizer differed. Each version ran the same 500 requests on an RTX 5090 with vLLM, and cost was calculated from measured time.

The first version merged each competency's score with the surrounding JSON into one macro-token. It was the most compact (73.3% lower cost), but quality dropped to 0.

689, against 0.703 for the production version. The score inside the macro turned every combination of score and structure into a separate token variant, and the model made the whole choice in a single step.

A separate token for the score cost two extra tokens per response, but it raised the score to 0.810, and the cost reduction reached 77.0%.

A test on broader material gave the following picture: five domains (contracts, medical papers, scientific papers, news, Python code) and three model families (Qwen3, Llama-3.1, Gemma-3). VocabX compressed more than the best reference in all 15 combinations, by 1.2–9.3 points. By estimated inference cost, the trimmed version also won everywhere: by 10.2 points on average on Qwen3, 7.5 on Llama-3.1 and 13.6 on Gemma-3. On contracts, it delivered 62.6% reduction with 113 426 vocabulary entries, while the best in-place BPE variant got 61.8% with 207 803.

When it is worth touching the tokenizer

The approach suits narrow production models with a repetitive response structure: graders, extractors, JSON generators, and legal and code assistants. The more template there is in the response and the more decode is bound by memory, the bigger the effect. The gain comes from three parts: a shorter response, a smaller LM head matrix and fewer reads at every step.

There is also a caveat. If the fragment being compressed contains the decision the model exists to make (here, the score), it must not be hidden inside a macro-token. The goal is the cheapest vocabulary that keeps the meaningful tokens. Without that, you can save on cost and lose quality.