DeepSeek Open-Sources DeepInfer: What It Takes to Run

§ 01What DeepSeek released, and what it kept#

The repository describes itself plainly: “DeepInfer is the high-performance inference framework that powers DeepSeek’s large-scale production services.” The README then lists what a complete service also needs and this release does not include.

DeepSeek gives its reason in the Q&A: “The KV cache management and load balancing components of the inference system are tightly coupled to our internal infrastructure, so that code is not yet fully open-sourced.” It describes their design in a Zhihu post and in the Q&A instead.

The engine supports one model, DeepSeek-V4.1-Flash, and two kinds of hardware: NVIDIA SM100 GPUs (the Blackwell data-center generation) and Huawei Ascend 950 NPUs. It needs Linux, Python 3.12, and CUDA 13 or CANN 9.2. The README credits vLLM as the original base, with ideas from SGLang and TensorRT-LLM. The thread promises more: “Inference code for future DeepSeek models and ongoing optimizations will land here too.”

§ 02The numbers DeepSeek disclosed#

The Q&A opens with the figure most readers will remember: “A single cluster currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at valley pricing).” Valley pricing is DeepSeek’s off-peak API rate, so that revenue figure is a floor, not an average.

The storage behind it is just as large. DeepSeek keeps roughly 100 PiB of SSD for KV cache, the computed context it reuses when a request repeats a prefix. Agent traffic is kept for 3 days and chatbot traffic for 7, because chatbot hits have a long tail: a small amount of cache is read very many times. In practice that works out to about 14.4 TB of logical cache per online GPU.

§ 03Why the floor is 128 GPUs#

DeepSeek splits the work in two. Prefill reads the prompt and runs on small groups of 2 to 8 GPUs. Decode writes the answer one step at a time and runs spread across 64 to 128 GPUs, each holding a slice of the model’s experts. In DeepSeek’s words, “the smallest efficient deployment unit is 64 GPUs of prefill plus 64 GPUs of decode.”

Spreading decode wide pays because each GPU moves less expert data per step. DeepSeek’s own chart compares 8 GPUs with 64 at the same load.

At 64 requests per rank, the wide setup gives each GPU about 24% more output and each user faster text, roughly 246 tokens per second against 197. The values are read off DeepSeek’s chart, measured with 131,072-token context and three draft tokens per step, so treat them as close, not exact.

§ 04Speculative decoding that checks fewer guesses#

DeepInfer decodes with a draft model that proposes 5 tokens at a time. DeepSeek’s DSpark scheduler then predicts how likely each draft token is to be accepted and verifies only the ones worth the compute. The Q&A says this cuts the tokens verified per step by about 30% while the accepted length barely moves.

DeepSeek runs production around 60 sequences per rank, where the gain is 4% to 7%. At light load the scheduling costs slightly more than it saves.

§ 05How close the public engine is to DeepSeek’s own#

DeepSeek says its internal kernel libraries are broadly the same as the public ones. The public FlashMLA lacks one hand-tuned optimization, which makes its decode kernel 5% to 10% slower, and the end-to-end gap is at most 5%. For a correctness check, the repository lists Pass@1 reference figures for V4.1: 99.9% on AIME 2026, 93.4% on GPQA Diamond and 42.6% on HLE-Full.

deepseek-recipe matters on its own. It encodes special tokens from their known positions, so text such as a user typing DeepSeek’s control tokens is encoded as ordinary text and cannot trigger a tool call. Third-party hosts of DeepSeek models get the same request handling as DeepSeek’s own API.

§ 06Should you self-host it?#

For most teams, no. You need Blackwell-class or Ascend 950 hardware, around 128 GPUs to run efficiently, and your own KV cache store and load balancers. For a cloud provider or a large inference team already serving DeepSeek V4.1 Flash, it is something rarer: a lab’s production engine, with the operating numbers attached.

§ 07Where CellCog stands#

CellCog does not serve DeepSeek models; every CellCog agent tier runs on Claude Opus 5.5. We follow inference releases like this one because they set what it costs to serve the models businesses run on.

§ 08What we are watching for#

• The KV cache and load-balancing code, which DeepSeek says is not yet fully open.
• Support for more models, which DeepSeek says will land in the same repository.
• Independent throughput numbers on hardware outside DeepSeek’s own clusters.

§ 09Sources#

• DeepSeek, DeepInfer repository, README and eval README, read October 20, 2026, 12:20 UTC.
• DeepSeek, DeepSeek V4.1 Flash Inference Deployment Q&A, October 20, 2026. Read directly.
• DeepSeek, deepseek-recipe repository, README. Read directly.
• DeepSeek on X, announcement thread, October 20, 2026, 09:30 UTC.