DeepSeek Open-Sources Internal Inference Engine DeepInfer: Supports Ascend 950, but Not Yet a Full Online Service

Beating AI News Flash: DeepSeek has officially open-sourced its internal inference framework DeepInfer under the Apache 2.0 license. The system has already been used to support large-scale online services for DeepSeek V4.1 Flash. Previously, DeepSeek had successively open-sourced the DeepGEMM computation library, the FlashMLA attention operator, and the DeepEP communication library. This time, it further disclosed the core framework responsible for multi-GPU parallelism, task scheduling, and generation acceleration.

DeepInfer aims to handle more requests with the same computing power while shortening user wait times. It can distribute different experts of the model across multiple compute cards and allow CPU scheduling and GPU computation to proceed simultaneously. DSpark speculative decoding can generate multiple candidate Tokens in advance and then schedule verification based on confidence. SWA Bounded Replay enables the model, when processing long texts, to recompute only a small recent segment of content, reducing redundant computation. In addition, it supports structured outputs such as JSON and reproducible sampling mechanisms.

There is also a twist behind this open-sourcing. In April 2025, DeepSeek explained that its internal inference engine was deeply modified from early vLLM and highly coupled with its own cluster, making it difficult to open directly to the community, so it decided to first extract independent components. Now the core code has finally been made public, but it is still not a complete online deployment solution. GitHub explicitly notes that the current version mainly provides offline inference and still lacks components such as KV cache management, separation of input and generation stages, and cross-replica load balancing.

Currently, DeepInfer only supports DeepSeek V4.1 Flash and is adapted for NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The official performance tests used 4 and 64 SM100 GPUs respectively to measure input processing and output generation, but did not provide same-condition performance comparisons with vLLM or SGLang. The simultaneously updated deepseek-recipe provides message and Token encoding/decoding capabilities aligned with the official API, supporting formats such as reasoning content, images, and tool calls.