KAIST drops calibration from MoE decoding, speeds up blocks by up to 2.69× over vLLM

Mixture-of-Experts decoding at small batch sizes is bound by memory, not compute: for every token, the model has to load the weights of the routed experts. KAIST AI researchers Geunhyeon Kim and Sung Ju Hwang show that a popular speedup technique, selecting important channels inside an expert, can work without calibration data. Their reader, built only on model weights, keeps 98.1% of the full model's accuracy while reading 17.8% of expert weights. MoE blocks on Triton kernels run 1.60–1.90 times faster than optimized vLLM at batch one.

The threshold promises a budget but does not keep it

The idea behind channel selection is simple. In each expert, most intermediate channels have little effect on a given token. A selector predicts which channels matter and reads only their weights from memory. The usual approach is to take channels whose score exceeds a threshold, and to tune that threshold on an offline sample.

The authors show that this scheme has two weaknesses. First, the threshold does not fix the read volume: the number of channels that pass the cutoff varies from input to input. Prox, with thresholds calibrated for an 18% budget, actually read 12.9% on average on their benchmarks. An engineer planning latency and throughput gets a different number from the one they budgeted for.

Second, an accuracy gain at a given budget is not guaranteed either.

Top-k instead of a threshold

The alternative needs no data at all: for each token, keep the k channels with the highest scores. The read budget is known in advance and matches the stated figure. Compared on identical scores, the top-k version of the selector matched or beat the threshold version in 40 of 43 cases at 13% and 18% budgets. On generation tasks, it won in all 23 comparisons.

There is also a qualitative effect. Calibrated thresholds sent Gemma 4 into repetition loops, while the top-k versions avoid this. The authors' conclusion: threshold calibration delivers neither of the two things it is meant to deliver, that is, neither a known read budget nor a consistent accuracy gain at that budget.

A reader built from weights alone

The second part of the work is the authors' own reader, which needs neither calibration data nor fine-tuning. It has two steps:

• channels are coarsely filtered using a ternary sketch of the weights;
• skipped channels are compensated by a low-rank mapping, which is also built from the model's weights.

Key figures across three post-trained MoE models:

• accuracy: 98.1% of the full model;
• volume of routed-expert weights read: 17.8%;
• speed of MoE blocks on Triton kernels at batch one: 1.60–1.90 times higher than optimized vLLM;
• at larger batch sizes, the speedup reaches 2.69 times on the same three models;
• Prox, calibrated for 18%, actually read 12.9%;
• top-k vs. threshold: 40 of 43 comparisons at 13% and 18% budgets, 23 of 23 on generation.

Who this is useful for

The biggest gains go to those running large MoE models on hardware with limited memory bandwidth. In a Hugging Face discussion, one of the authors called activation pruning an acute problem for GPUs with slow memory, such as DGX Spark. There, memory bandwidth, not FLOPS, determines token generation speed. According to him, the next form of sparsity will come precisely from activation sparsity when serving on devices with a hard bandwidth limit.

The practical advantage of dropping calibration is that the method can be applied to any new open-weights model right after release. There is no need to find a representative sample, and no risk that a threshold tuned on one domain will break generation on another, as happened with Gemma 4. A fixed read budget simplifies planning: the 17.8% built into the design translates into predictable savings in memory traffic and latency.

The code is published in the calibration-free-moe repository on GitHub. For teams that write their own Triton kernels or optimize local inference, it is a ready starting point. It is worth testing on your own models, especially in interactive batch-one scenarios, where memory is the tightest bottleneck.