# DeepSeek Open-Sources DeepInfer: What It Takes to Run

> DeepSeek open-sourced DeepInfer, the engine behind V4.1 Flash. What is in the repo, what is held back, and why the efficient floor is 128 GPUs.

- Author: Nitish Garg, Founder & CEO, CellCog
- Published: 2026-10-20
- Canonical (HTML): https://cellcog.ai/blog/deepseek-deepinfer-open-source/
- Section: Guides / Choosing a platform
- Publisher: CellCog (https://cellcog.ai), the AI employee platform. Blog index for agents: https://cellcog.ai/blog/llms.txt

## Key points

- DeepSeek open-sourced DeepInfer, the inference engine behind DeepSeek V4.1 Flash, under Apache 2.0 on October 20, 2026, with deepseek-recipe, its request-to-token converter.
- DeepSeek says one cluster serves 155 trillion tokens a day and earns about 38 dollars per GPU-hour at its off-peak (valley) price.
- deepseek-recipe turns Chat Completions, Responses and Messages requests into DeepSeek prompts and parses output back, matching the official API's behavior.
- It runs one model, DeepSeek-V4.1-Flash, on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs; the efficient minimum is 64 prefill plus 64 decode GPUs.
- The KV cache system, prefill and decode disaggregation, and load balancing are not in the release; DeepSeek says that code is tied to its internal infrastructure.

## At a glance

- **What is DeepInfer?** DeepSeek's own inference engine, the software that serves DeepSeek V4.1 Flash to its API users, now public on GitHub under Apache 2.0.
- **Can I run DeepSeek's whole service with it?** No. The engine is there, but the KV cache system, the split of prefill and decode onto separate machines, and the load balancers are not. You build those yourself.
- **How big a deployment does it expect?** DeepSeek's own guide puts the smallest efficient unit at 64 GPUs for prefill plus 64 for decode, on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs.

**DeepSeek has open-sourced DeepInfer, the inference engine that serves DeepSeek V4.1 Flash in production, under the Apache 2.0 license.** It announced the release on X at 09:30 UTC on October 20, 2026, together with deepseek-recipe, the component that turns API requests into tokens and back. By noon UTC the repository had more than 6,400 stars and 500 forks.

The code comes with a long deployment Q&A that says more than the README: one DeepSeek cluster serves 155 trillion tokens a day, the efficient minimum is 128 GPUs, and the parts that make it a full service are not in the box. This post reads the repository, the Q&A and DeepSeek's benchmark charts directly.

## What DeepSeek released, and what it kept

The repository describes itself plainly: "DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services." The README then lists what a complete service also needs and this release does not include.

*Table: What is in the DeepInfer release (DeepSeek README and deployment Q&A, October 20, 2026)*

| Part | In the release? | What it does |
|---|---|---|
| DeepInfer engine | Yes, Apache 2.0 | Kernels, model and sampler runtime, scheduling, speculative decoding, structured outputs |
| deepseek-recipe | Yes | Converts Chat Completions, Responses and Messages requests to DeepSeek V4 and V4.1 prompts, and output back |
| KV cache system | No | Stores computed context on a dedicated 3FS storage cluster so repeated prefixes skip prefill |
| Prefill and decode disaggregation | No, though most engine pieces are in | Runs vision encoding, prefill and decode on separate machines |
| Load balancing | No | Spreads requests across data-parallel replicas for prefill and decode |

![Animation: a request flows through DeepSeek's serving system; DeepSeek Recipe and the DeepInfer engines in prefill and decode turn teal under an Apache 2.0 tag, while load balancers and KV cache storage stay grey with padlocks, marked not in the release](https://cellcog.ai/blog/media/deepseek-deepinfer-open-source/deepinfer-open.svg)
*The engine and request converter are open; KV cache storage and load balancers are not*

DeepSeek gives its reason in the Q&A: "The KV cache management and load balancing components of the inference system are tightly coupled to our internal infrastructure, so that code is not yet fully open-sourced." It describes their design in a Zhihu post and in the Q&A instead.

The engine supports one model, DeepSeek-V4.1-Flash, and two kinds of hardware: NVIDIA SM100 GPUs (the Blackwell data-center generation) and Huawei Ascend 950 NPUs. It needs Linux, Python 3.12, and CUDA 13 or CANN 9.2. The README credits vLLM as the original base, with ideas from SGLang and TensorRT-LLM. The thread promises more: "Inference code for future DeepSeek models and ongoing optimizations will land here too."

## The numbers DeepSeek disclosed

The Q&A opens with the figure most readers will remember: "A single cluster currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at valley pricing)." Valley pricing is DeepSeek's off-peak API rate, so that revenue figure is a floor, not an average.

The storage behind it is just as large. DeepSeek keeps roughly 100 PiB of SSD for KV cache, the computed context it reuses when a request repeats a prefix. Agent traffic is kept for 3 days and chatbot traffic for 7, because chatbot hits have a long tail: a small amount of cache is read very many times. In practice that works out to about 14.4 TB of logical cache per online GPU.

## Why the floor is 128 GPUs

DeepSeek splits the work in two. Prefill reads the prompt and runs on small groups of 2 to 8 GPUs. Decode writes the answer one step at a time and runs spread across 64 to 128 GPUs, each holding a slice of the model's experts. In DeepSeek's words, "the smallest efficient deployment unit is 64 GPUs of prefill plus 64 GPUs of decode."

Spreading decode wide pays because each GPU moves less expert data per step. DeepSeek's own chart compares 8 GPUs with 64 at the same load.

![Animation: 8 large GPUs crowded with experts put out about 12,700 tokens per second each; 64 small GPUs holding one or two experts each put out about 15,700](https://cellcog.ai/blog/media/deepseek-deepinfer-open-source/deepinfer-wide-ep.svg)
*Spread over 64 GPUs, each holds fewer experts and puts out more tokens per second*

At 64 requests per rank, the wide setup gives each GPU about 24% more output and each user faster text, roughly 246 tokens per second against 197. The values are read off DeepSeek's chart, measured with 131,072-token context and three draft tokens per step, so treat them as close, not exact.

## Speculative decoding that checks fewer guesses

DeepInfer decodes with a draft model that proposes 5 tokens at a time. DeepSeek's DSpark scheduler then predicts how likely each draft token is to be accepted and verifies only the ones worth the compute. The Q&A says this cuts the tokens verified per step by about 30% while the accepted length barely moves.

*Table: Fixed-length vs dynamic verification, EP64 benchmark (DeepSeek deployment Q&A)*

| Sequences per rank | Fixed-5 throughput (tok/s/rank) | Dynamic throughput (tok/s/rank) | Gain |
|---|---|---|---|
| 16 | 4,891 | 4,816 | -1.5% |
| 32 | 8,964 | 9,111 | +1.6% |
| 48 | 12,032 | 12,467 | +3.6% |
| 60 | 13,730 | 14,466 | +5.4% |
| 72 | 14,962 | 15,961 | +6.7% |
| 96 | 16,817 | 18,660 | +11.0% |

DeepSeek runs production around 60 sequences per rank, where the gain is 4% to 7%. At light load the scheduling costs slightly more than it saves.

## How close the public engine is to DeepSeek's own

DeepSeek says its internal kernel libraries are broadly the same as the public ones. The public FlashMLA lacks one hand-tuned optimization, which makes its decode kernel 5% to 10% slower, and the end-to-end gap is at most 5%. For a correctness check, the repository lists Pass@1 reference figures for V4.1: 99.9% on AIME 2026, 93.4% on GPQA Diamond and 42.6% on HLE-Full.

deepseek-recipe matters on its own. It encodes special tokens from their known positions, so text such as a user typing DeepSeek's control tokens is encoded as ordinary text and cannot trigger a tool call. Third-party hosts of DeepSeek models get the same request handling as DeepSeek's own API.

## Should you self-host it?

For most teams, no. You need Blackwell-class or Ascend 950 hardware, around 128 GPUs to run efficiently, and your own KV cache store and load balancers. For a cloud provider or a large inference team already serving DeepSeek V4.1 Flash, it is something rarer: a lab's production engine, with the operating numbers attached.

## Where CellCog stands

CellCog does not serve DeepSeek models; every CellCog agent tier runs on Claude Opus 5.5. We follow inference releases like this one because they set what it costs to serve the models businesses run on.

## What we are watching for

- The KV cache and load-balancing code, which DeepSeek says is not yet fully open.
- Support for more models, which DeepSeek says will land in the same repository.
- Independent throughput numbers on hardware outside DeepSeek's own clusters.

## Sources

- DeepSeek, [DeepInfer repository](https://github.com/deepseek-ai/DeepInfer), README and eval README, read October 20, 2026, 12:20 UTC.
- DeepSeek, [DeepSeek V4.1 Flash Inference Deployment Q&A](https://github.com/deepseek-ai/DeepInfer/blob/main/doc/inference_deployment_qa.md), October 20, 2026. Read directly.
- DeepSeek, [deepseek-recipe repository](https://github.com/deepseek-ai/deepseek-recipe), README. Read directly.
- DeepSeek on X, [announcement thread](https://x.com/deepseek_ai/status/2112476496283775135), October 20, 2026, 09:30 UTC.

## FAQ

**When was DeepInfer released?**

DeepSeek announced it on X at 09:30 UTC on October 20, 2026. The repository's first commit is dated 04:09 UTC the same day.

**What license is DeepInfer under?**

Apache License 2.0, per the repository and its license file. deepseek-recipe was released at the same time.

**Which models and hardware does it support?**

One model, DeepSeek-V4.1-Flash. Hardware: NVIDIA SM100 GPUs with CUDA 13, or Huawei Ascend 950 NPUs with CANN 9.2, on Linux with Python 3.12.

**Is the public version as fast as DeepSeek's internal one?**

Close, per DeepSeek's Q&A: the public FlashMLA lacks one hand-tuned optimization that makes its decode kernel 5% to 10% slower, and end-to-end performance is at most 5% apart.

**What scores should a correct deployment reproduce?**

DeepSeek lists Pass@1 reference figures of 99.9% on AIME 2026, 93.4% on GPQA Diamond and 42.6% on HLE-Full, as a sanity check rather than exact targets.

**Does CellCog run DeepSeek models?**

No. Every CellCog agent tier runs on Claude Opus 5.5. We follow inference releases because they set what it costs to serve the models businesses depend on.

## Related

- [DeepSeek V4.1 Flash: Price, Specs, and the Rumor Scorecard](https://cellcog.ai/blog/deepseek-v4-1-flash-release-date/index.md)
- [What Is DeepSeek Harness? The Open Agent, Explained](https://cellcog.ai/blog/what-is-deepseek-harness/index.md)
- [DeepSeek V5 Release Date: The Leak vs the Record](https://cellcog.ai/blog/deepseek-v5-release-date/index.md)

## The AI employee for this read

[AI Software Engineer](https://cellcog.ai/ai-employees/ai-software-engineer): I built this page. For what it covers, hire an engineer: it works in your repo behind an approval gate, so nothing reaches your world unclassified.

---

Markdown alternate of https://cellcog.ai/blog/deepseek-deepinfer-open-source/. Try CellCog free, no credit card needed: https://cellcog.ai/signup