DeepSeek open-sources DeepInfer: 155 trillion tokens a day and an 80% margin

DeepSeek has released DeepInfer, the engine that runs V4.1 Flash, in open access. The code comes with an engineering report containing production numbers: 155 trillion tokens a day from a single cluster, a 96% input cache hit rate, an average time to first token of 400 ms, and generation at 230 tokens/s (up to 300 for code). Latent Space calculates that, at special prices of $0.15 per million input tokens and $0.60 per million output tokens, this comes to about $22 million in revenue per day and roughly $38 per accelerator per hour.

Where the 80% margin comes from

The arithmetic goes like this. 155 million million tokens at a blended price of about $0.14 yield roughly $1 million per hour. The analysis author estimates the full cost of a card at $5–8 per hour. That gives an inference margin above 80% and a cluster payback period of about 81 days. This is Latent Space's estimate; DeepSeek itself plans for payback in 10 months.

An important caveat: 155 trillion served tokens are not the same as 155 trillion generated tokens. With a 96% cache hit rate, almost all input comes from storage, and only a small share is recomputed. The figures are first-party, there has been no independent audit, and the result is sensitive to workload mix, cache accounting and which costs are included.

How the stack works

DeepInfer separates vision encoding, prefill and decode, and scales each stage according to its own bottleneck. Prefill is recommended to run on ordinary eight-accelerator machines, and decode on supernodes of 64–128 accelerators with wide expert parallelism for MoE.

• The prefill scheduler estimates attention cost and cuts chunks to fit a time budget rather than a fixed size.
• Prefill routing is not tied to the cache: prefixes sit in shared storage, and a request goes to any free worker.
• Decode routing takes into account the number of active requests and the KV-cache length.
• Popular experts are replicated and periodically redistributed, at the cost of HBM memory.
• The CPU scheduler runs asynchronously and removes the pauses between decode steps.
• According to the analysis, the disclosed Blackwell configuration does not use tensor parallelism.

The cache sits on a 3FS cluster with roughly 100 PiB of SSD. Data is retained for 3 days for agents and 7 for chats. Sliding-window attention state stays in host memory. The SWA Bounded Replay method restores the missing short-lived state from the last 128 tokens and, according to DeepSeek, cuts prefill computation by almost half. DSpark, an adaptive speculative verification scheme, adds +6% to production throughput relative to a fixed check of five draft tokens.

The deepseek-recipe component handles the interface layer: Chat Completions, Responses and Messages APIs, parallel tokenization, and the distinction between genuine control tokens and identical strings in user text. It is not full protection against prompt injection.

What is open and what is not

The team calls the release fully open, but the technical description indicates that KV-cache management and the load balancer have not yet been fully published. They are tied to internal infrastructure. 3FS, released earlier, is part of the picture too. Service-level orchestration remains the job of whoever deploys the stack.

Community estimates are secondhand: 196B Engram parameters, 552B expert parameters, DP64/EP64, mostly 4-bit weights, and 400x KV compression relative to GQA. The primary source does not confirm these details.

vLLM responded with benchmarks. DeepInfer was originally built on vLLM, and the project team showed decode throughput within 6% of DeepSeek's published figure on GB200 in a comparable configuration, and published recipes and AgentX results. This is parity for a single stage, not a reproduction of the whole service: the 100 PiB cache, routing and availability are not part of that result.

At the same time, OpenAI announced 12-hour prompt caching in its API, instead of the default 30 minutes. The products are different, but the goal is the same: agents that return to a long context hours later.

A team building an agent service should take three things from this release. First, calculate economics from the cache hit share and prefix retention time, not from the price per token. Second, split prefill and decode into separate pools and balance prefill by estimated compute. Third, test vLLM on your own workload, because its decoding is already close to DeepSeek's results, and the main gap lies in the cache and orchestration layer.