DeepSeek Open-Sources DeepInfer, the Framework Running Its 155-Trillion-Token-a-Day Cluster

DeepSeek released the source code for DeepInfer, the inference framework that runs its own production services, on October 20, 2026. The repository is published on GitHub under the Apache 2.0 license and currently supports the company's DeepSeek-V4.1-Flash model, specifically the 1017 version by default, according to the project's own documentation.

DeepInfer is not a new model. It is the serving layer that sits between a trained model and the users sending it requests — the software that decides how prompts are batched, how computation is split across GPUs, and how tokens are generated and returned. DeepSeek describes it in the repository as providing the core features needed for serving large models at scale, though it says a complete production service also requires additional components it has not released.

The release documents, rather than invents, the stack DeepSeek already runs. Reporting from SiliconANGLE, drawing on DeepSeek's own deployment documentation, says one of the company's clusters now serves 155 trillion tokens a day, and that revenue at off-peak pricing works out to $38 per GPU per hour. DeepSeek had previously held its full serving code back; SiliconANGLE notes the company said in April 2025 that maintaining a large open-source project was more than "a small research team focused on developing better models" had the capacity for.

What is actually inside the repository

The GitHub documentation lists a set of custom kernels — FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect — along with TileLang operations for the low-level compute work. For generation speed, DeepInfer uses a speculative-decoding system called DSpark, which accelerates decoding with draft tokens and uses confidence estimates to schedule verification, so a smaller draft model proposes several tokens at a time and a confidence head determines which ones the main model is likely to accept, spending verification compute on the likeliest candidates rather than on every token. SiliconANGLE reports that DeepSeek's own documentation puts the throughput gain from this approach at 4% to 7% at production load, and describes a separate optimization — reducing how many tokens pass through the full decoder during the prefill stage — that the company says nearly halves total prefill computation.

Hardware support is split two ways: Nvidia's SM100 ("Blackwell") GPUs and Huawei's Ascend 950 NPUs. Older Nvidia architectures, including Hopper-generation chips, are not supported in this release, and deterministic sampling — reproducible outputs for identical inputs — is currently limited to the Nvidia path, per the repository's documentation. The framework also supports batch-invariant inference and structured outputs constrained by regex, JSON schemas or EBNF grammars, features relevant to anyone building agents or tools that need predictable, parseable model output.

What was left out

The repository documentation states plainly that several pieces of DeepSeek's production stack are not included: key-value cache management, the infrastructure for disaggregating prefill and decode across separate machine pools, and the load-balancing logic that spreads traffic across data-parallel replicas. DeepSeek attributes the omission to how tightly those systems are coupled to its internal infrastructure. SiliconANGLE also notes that DeepInfer builds on vLLM, the widely used open-source inference engine, as its original foundation, though neither DeepSeek's documentation nor the SiliconANGLE report offers a side-by-side comparison of DeepInfer against vLLM or the other major open-source serving engine, SGLang.

Why it matters for researchers and engineers

For anyone outside DeepSeek who wants to serve its models at scale, DeepInfer is a chance to see the specific kernel and scheduling choices a frontier lab makes under real production load, including the Pareto curves the repository publishes showing throughput against latency on 4 SM100 GPUs for prefill and 64 for decode. The repository states that the smallest deployment the company considers efficient uses 64 GPUs for prefill and another 64 for decoding, which puts the framework's practical floor well above what a university lab or small engineering team typically has on hand; it is built for scale, not for running a single model on a workstation.

There is no independent benchmark yet comparing DeepInfer's throughput or latency claims against vLLM, SGLang or other serving engines on equivalent hardware; the figures in circulation — the 155-trillion-token daily figure, the $38-per-GPU-hour estimate, and the 4% to 7% speculative-decoding gain — all trace back to DeepSeek's own documentation rather than to a third-party evaluation. Students and engineers studying inference-serving design now have a working reference implementation from a lab operating at genuine production scale, but the missing cache-management and load-balancing code means it is not yet a complete, drop-in replacement for the parts of the stack DeepSeek has kept to itself.