Cline squeezes 11 840 tokens/s out of a single NVIDIA DGX Station

The Cline team raised the aggregate generation speed of an open model on a single GPU to 11 840 output tokens per second. The record was set on October 15 on an NVIDIA station that the engineers could access thanks to their collaboration with NVIDIA. The starting point was already high: plain vLLM 0.32.0rc1 delivered 6 058 tokens/s on a test with 512 input and 512 output tokens at 256 concurrent streams. The final result is about 1.95 times higher.

The model was built as a fast local workhorse for a range of tasks: document parsing, coding, financial accounting. On a single machine it has a GPU memory budget of 252 GB. In Cline's demo on such a station, with no video acceleration, 32 parallel Cline instances run on top of Nemotron 3.5 Super VL.

Five steps from 6 058 to 11 840

Cline describes not a single discovery but a series of experiments, of which five worked:

• Built-in MTP head. Multi-token prediction shipped with the model. The engineers started with MTP3, meaning three draft tokens per step, and pinned OMP1: a single CPU thread works better because multiple threads add coordination overhead.
• Asynchronous scheduling. At high concurrency, the GPU is busy with compute kernels while the CPU is busy with scheduling. Async mode lets the scheduler avoid waiting for the GPU to finish its work.
• Mamba kernel from FlashInfer. The model is a hybrid: attention layers use a KV-cache, while Mamba layers store a recurrent state and a short convolution history. Switching back to the FlashInfer kernel produced a noticeable jump. The authors did not fully work out why, and the effect may not repeat on other hardware.
• MTP1 instead of MTP3. A counterintuitive step that raised throughput again.
• Mamba state in FP16 instead of FP32 with stochastic rounding. This lowers precision and could potentially hurt benchmark metrics, but for the goal of maximum tokens per second the cost turned out to be acceptable.

Why fewer draft tokens turned out faster

Speculative decoding looks like a free bonus: the more draft tokens predicted, the better the chance of accepting several at once. But every draft has to be verified. If verification fails, the model's time is wasted. At 256 concurrent streams, the GPU is already full of useful work, and the extra verification does not pay off. That is why MTP1 beat MTP3.

MTP's benefit can look opposite at low load and at high concurrency. The same recipe could hurt on a different traffic profile. Cline says outright that inference engineering is highly context-dependent, and that scaling depends on the shape of the traffic.

Economics: 66% off the API price

Cline's main practical takeaway comes from a back-of-the-envelope calculation. Compare paying for an API with running a fully loaded GPU, and the result is a 66% discount on inference. The condition is strict: the GPU must be busy almost all the time. In that case it is cheaper to take Cline's recipe and pay for the hardware than to go through a model router and pay for every call.

Two predictions from the team follow from this. First: self-hosting dozens of parallel local agents will become routine practice, and 32 Cline instances on one station already show it. Second: inference autoscaling will become agentic. An agent with access to Grafana metrics will be able to run load tests on a dedicated GPU that does not touch production traffic. It will pick the best configuration for the real shape of requests, and then carefully shift traffic to the new configuration. Such a "self-healing" system will adjust GPU consumption to the time of day and the load.

What to check before repeating the recipe

The recipe should not be copied blindly. First, pin down your own profile: prompt and response lengths, the number of concurrent streams, acceptable latency. Next, it is worth running the same five switches one at a time. For high concurrency, compare MTP1 and MTP3; with a small number of streams, the effect may turn out to be the opposite. Replacing FP32 with FP16 in the Mamba state absolutely requires re-running your evaluation sets, because the speed here is bought at the cost of precision. Finally, the 66% savings estimate holds only with high GPU utilization: if the station sits idle, the API becomes the better deal again.