DeepSeek just did something no frontier AI lab has done before: it open-sourced the software that serves its models to the world. On October 20, 2026, the Chinese lab released DeepInfer — its in-house inference framework — on GitHub under the Apache 2.0 license. Inference is the business of running a trained model for users, and DeepInfer is the engine DeepSeek says powers its own production services — behind its famously cheap AI.
DeepInfer is built primarily on vLLM, with ideas borrowed from SGLang and NVIDIA's TensorRT-LLM. The secret sauce is a set of custom kernels — FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and TileLang — tuned in-house. A companion release, deepseek-recipe, adds an MIT-licensed tokenizer matching DeepSeek's official API.
What's inside DeepInfer
Hardware support is deliberately narrow: only NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs work — older NVIDIA cards like Hopper are out. For now, one model runs on it, DeepSeek-V4.1-Flash (the updated 1017 build), and only in offline inference mode.
The performance numbers are DeepSeek's own claims, not independently verified. The company says one DeepInfer cluster serves 155 trillion tokens per day with a 96% input cache hit rate, about 400ms time-to-first-token, and 230 tokens per second — up to 300 for coding. At off-peak pricing, it claims about $38 of revenue per GPU-hour.
DSpark speculative decoding has a draft model propose five tokens at a time, with a confidence head predicting which the main model will accept — about 30% fewer tokens checked, 4–7% more throughput. A prefill trick nearly halves prefill compute. Also included: batch-invariant inference (identical inputs, identical outputs) on NVIDIA hardware, and structured outputs via regex, JSON schema and EBNF.
Notably absent: KV-cache management and load-balancing code, which DeepSeek says are too tightly coupled to its internal infrastructure. Splitting prefill and decode across machines is left to operators.
Why this is a big deal
The reversal is striking. In April 2025, DeepSeek said it would not release the full engine. Now it has — the first time a frontier lab has handed out its actual production serving stack, the kind of software OpenAI and Anthropic keep locked down. That puts pressure on managed-inference providers, whose business is selling exactly this capability.
The Ascend 950 support is a first-party production endorsement of Huawei's hardware, arriving as DeepSeek plans to deploy 160,000 Ascend 950DT accelerators in Inner Mongolia, per Bloomberg's September reporting. Timing is pointed: DeepInfer landed a day after the US Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products.
The catches
This is no download-and-run deal: offline-only, one model, data-center-class hardware, deterministic features on NVIDIA only — and every benchmark above is DeepSeek's own unverified claim. Still, it extends the open weights playbook: rivals release open models, while DeepSeek open-sources the machinery that serves them cheaply.
Verdict
DeepInfer won't change your chatbot today. But it resets the economics of serving AI at scale — arguably the most generous infrastructure release from a frontier lab this year.
Comments
Post a Comment