DeepSeek-AI has published DeepInfer as a public GitHub repository under the Apache License 2.0. The project is presented as the inference framework behind DeepSeek's high-performance production services. This release focuses on offline inference and the engine capabilities needed for large deployments. The repository page showed 5.3k stars and 398 forks.

terminal
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.py

Only DeepSeek-V4.1-Flash is listed as supported. Parallel execution covers Attention DP and balanced CP. MoE workloads use EP + EPLB. Decode acceleration combines DSpark with confidence-scheduled verification, which uses draft tokens and confidence estimates to schedule checks.

The runtime includes SWA bounded replay for decoder prefill and an asynchronous batch scheduler. Batch-invariant inference and deterministic sampling round out the feature set. The scheduler is enabled by default. Batch-invariant inference requires DEEPINFER_BATCH_INVARIANT=1 and is limited to NVIDIA. Deterministic sampling additionally requires DEEPINFER_ENABLE_MTP_VARLEN=0 and a fixed seq_seed in SamplingParams.

The kernel layer incorporates FlashMLA, DeepGEMM and DeepEP. It adds TileKernels, DeepSelect and custom TileLang kernels. DSpark uses DEEPINFER_ENABLE_MTP_VARLEN=1, and DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length, which defaults to 5. SWA replay uses DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 for the default V4.1 settings. The retained suffix must cover the SWA window and the maximum backtracking length.

Generated output can be constrained by a regular expression. JSON Schema and EBNF grammar definitions are supported through so_args={"schema_type": ..., "schema": ...} in SamplingParams.

DeepInfer supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Building requires Linux, GCC >= 13.0 and Python 3.12. NVIDIA deployments need CUDA 13; Ascend deployments need CANN 9.2. The project asks users to create an empty Python 3.12 environment, set DEEPINFER_ACCELERATOR to nvidia or ascend, and run build.sh.

The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is downloaded to the Hugging Face cache on the first run. A local copy can be selected with DEEPINFER_HF_MODEL_DIR. Offline inference starts through python examples/offline_inference.py. The performance measurements used 4 SM100 GPUs for prefill and 64 for decode, with each test inside one scale-up domain. run_sweep.sh reproduces those benchmarks, and evaluation instructions are in eval/README.md.

A complete inference service still needs a KV cache system. VE/Prefill/Decode disaggregation and load balancing across data-parallel replicas for both prefill and decode are not included. The repository says most of the required engine functionality is already implemented. DeepInfer was built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. Its citation identifies DeepSeek-AI as the author and 2026 as the publication year.