Cline Pushes an Open Model to 11,840 Tokens per Second on a Single GPU

A strong starting point

On October 15, the Cline team published a report on how they pushed an open language model to a combined 11,840 tokens per second on a single GPU. The model in question is a compact design built specifically for local deployment — it's not aiming to be the smartest model around, but it needs to run fast and reliably across several tasks at once: document parsing, coding, financial calculations. That speed and efficiency is precisely what makes models like this appealing — they can be fine-tuned for a company's specific needs and run on in-house hardware instead of paying per-call for someone else's API.

The experiment was made possible through a partnership with NVIDIA, which gave Cline access to a powerful server equipped with a single GPU and 252 GB of video memory. The goal was simple: squeeze as many tokens per second as possible out of that setup under heavy parallel load.

Right out of the gate, running vLLM engine version 0.32.0rc1.dev190+g3f1bcfd82 with out-of-the-box settings, the model delivered 6,058 tokens per second on a benchmark with 512 input and 512 output tokens across 256 simultaneous request streams. That came as a bit of a surprise to Cline — the model's default configuration turned out to already be well-tuned for high load, and the team treated that number as a solid starting point rather than a failure in need of a heroic fix.

Five steps that nearly doubled throughput

From there, the engineers ran a series of experiments, changing one parameter at a time:

• Multi-token prediction (MTP3). The model ships with a head capable of predicting several upcoming tokens at once. Enabling MTP3 mode with single-threaded execution (OMP1) — to avoid the overhead of coordinating across threads — delivered the first speed gain.
• Asynchronous scheduler. Under heavy parallel load, the GPU stays busy computing while the CPU handles task scheduling. Switching the scheduler to asynchronous mode eliminated GPU idle time spent waiting on the CPU.
• Replacing the Mamba kernel with FlashInfer. The model uses two types of layers: standard attention layers with a KV cache, and Mamba layers, which store state differently — through recurrent memory and a short convolution over history. Switching the compute engine for the Mamba layers to FlashInfer unexpectedly produced a noticeable performance jump. The team is upfront that they don't fully understand why this worked — the effect may be tied to this specific GPU configuration and might not reproduce on different hardware.
• Reducing MTP from 3 to 1. It would seem that predicting more tokens ahead of time should speed up generation. In practice, the opposite happened: every prediction still has to be verified, and a failed verification wastes time. With 256 parallel streams already keeping the GPU fully occupied with useful work, the extra prediction bets just got in the way. Dropping the prediction depth to MTP1 delivered another speed boost.
• Mamba state in FP16. The final step was converting the internal state of the Mamba layers from 32-bit to 16-bit precision with stochastic rounding. This lowers computational precision and could slightly degrade response quality, but for the goal of this experiment — maximum speed — the team considered the trade-off justified.

Together, these five changes raised aggregate throughput from 6,058 to 11,840 tokens per second — nearly double, with no hardware swap involved.

What this means for engineers

The Cline team's main takeaway isn't a recipe so much as a warning: inference engineering is highly context-dependent. The tricks that worked here — MTP1 instead of MTP3, FlashInfer instead of the standard Mamba kernel — could just as easily slow things down under a different traffic shape (a different input/output token ratio, a different number of parallel requests). That's why Cline advises against blindly copying someone else's configuration — instead, teams should first understand the shape of their own traffic and tune inference accordingly.

The team also offered three predictions for the future. First, running dozens of parallel agentic workflows locally will become routine — they point to a demo showing 32 simultaneous Cline sessions on the Nemotron 3.5 Super VL model, running on a single NVIDIA DGX station, recorded without video acceleration.

Second, when a GPU is fully utilized, self-hosting models can end up cheaper than paying for API calls — Cline's own calculations put the savings at around 66% in their case. Third, the authors believe that finding the optimal inference configuration will itself become a task for AI agents over time: an agent with access to Grafana metrics could run load tests on a spare GPU, find the best configuration for a given traffic pattern, and safely switch production over to it — turning inference autoscaling into an agentic task.