DeepSeek DeepInfer: Open-Source SOTA Inference Running 155T Tokens/Day/Cluster, Making $22M/Day at ~80% Margin for an 81-Day Payback

Proudly brought to you by SerpAPI: Today’s release covers a LOT of ground, but you can scrape all this into markdown for your agents ! 50% token cost of JSON. We last covered the Inference Inflection in April , and historically most of the advances have been closed source decacorns from Fireworks to Baseten to Modal . Well, time for the open source response. It is day 1 of PyTorch Conf , and DeepSeek timed an intercontinental ballistic missile into the discussion by open sourcing DeepInfer , their production stack, capable of up to 72 hour prompt caching to the 12 hours OpenAI just announced: This is their customization of vLLM with credit to SGLang and TRRTLLM , but there is a lot added… minus their crown jewels - their ultra optimized KV cache (compact + >8hour lifetime), PD/disggregation, and load balancing (their 3FS was open sourced a while ago). We have translated their blogpost and published it here together with a complete research guide and explainers and codemap and subsystem guide : As we point out in their additional disclosure , their claim to serve 155T tokens/day gets them $38 per SM100/A950 card per hour at off peak prices (currently $0.15/$0.60 per million input/output tokens: 155000000 tokens * $0.14 blend cost / 24 hours is ~$1m per hour). Taking their production data (which we translated ) of 1.5 million cache-inclusive input tokens/second per prefill GPU, 96% input cache hit, and 14,000 output tokens/second per decode GPU (over ~2400 GPUs): with all-in costs $5-8 per card per hour that is an >80% inference margin ! Doing some cluster math, that is roughly $166B per GW, with 81 day payback period on inference margin (DeepSeek famously plans for 10 month payback). Imagine what the Whale could do with more GWs. Sponsored by SerpAPI! Give Your AI Agents Live Search Data with Fewer Tokens Receive results from any of SerpApi’s 100+ APIs as clean, structured Markdown , using roughly 50% fewer tokens than JSON on average. Simply add output=md or use the /search.md endpoint. It requires zero configuration and is included with every plan at no additional cost. AI News for 10/19/2026-10/20/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space . You can opt in/out of email frequencies! AI Twitter Recap Top Story: DeepSeek DeepInfer—Production Inference, Open-Source Boundaries, and the Economics of Cached Tokens What happened DeepSeek released DeepInfer, the inference engine behind V4.1 Flash, alongside production-serving details that make the engineering report as important as the code. The release covers the inference engine and API-processing layer. DeepInfer supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output on NVIDIA and Huawei Ascend platforms. The accompanying deepseek-recipe component implements production token encoding/decoding and official-API-compatible processing. DeepSeek says future model inference implementations and optimizations will also appear in this repository. @deepseek_ai , @deepseek_ai , announcement translation The reported production figures are substantial—but describe a heavily cached service. DeepSeek’s engineering report, as summarized and quoted in the supplied posts, reports 155 trillion tokens/day from one cluster , 96% input-cache hits , 400 ms average time to first token , and 230 tokens/s generation , reaching 300 tokens/s for coding . It also reports $38 revenue per accelerator-hour at off-peak prices . These are first-party production claims, not independently audited measurements. @ZhihuFrontier , @swyx “Fully open source” does not mean the entire production service is reproducible from this release. Although DeepSeek describes DeepInfer as fully open source, the technical account says the KV-cache management and load-balancing components are not yet fully released , because they remain coupled to internal infrastructure. The engine is open; important service-level orchestration remains a deployment responsibility. @deepseek_ai , @ZhihuFrontier vLLM supplied the most concrete comparative response. After acknowledging that DeepInfer was initially built primarily on vLLM and released under Apache 2.0 , the vLLM team reported decode throughput within 6% of DeepInfer’s published results on GB200 under a matched configuration. It also published AgentX results and reproduction recipes, materially qualifying claims that DeepSeek’s efficiency is inaccessible to existing open-source engines. @vllm_project , @vllm_project , reproduction resources The release triggered two distinct debates: whether DeepSeek has a uniquely strong serving stack, and whether its disclosures demonstrate high inference profitability more generally. The first now has comparative benchmark evidence; the second remains sensitive to workload mix, cache accounting, hardware cost, and which expenses are included. @Tim_Dettmers , @zephyr_z9 , @Yuchenj_UW Architecture: separate stages, schedule compute rather than tokens DeepInfer’s disclosed architecture separates vision encoding, prefill, and decode , allowing each stage to scale against its own bottlenecks. The recommended deployment shapes are ordinary eight-accelerator machines for prefill and 64–128-accelerator supernodes for decode . Decode uses wide expert parallelism to distribute MoE experts while maintaining large effective batches. These are deployment recommendations, not sufficient evidence of the exact hardware composition of DeepSeek’s production cluster. @ZhihuFrontier The scheduling details explain why a conventional “tokens per batch” comparison can miss substantial differences: Prefill scheduling estimates attention-compute cost. Work is allocated using estimated compute rather than raw request length alone. Chunk sizes vary to fit a time budget, instead of enforcing one fixed chunk size across heterogeneous requests. @ZhihuFrontier Prefill routing does not require cache affinity. Because cached prefixes are pooled in shared storage, a request can go to an available worker rather than the worker that previously served its prefix. In a detailed Chinese-language analysis, inference engineer Su Jin Yan highlights this as a particularly useful production design: prefill data-parallel groups balance on “compute tokens,” while shared storage supplies reusable state. The same analysis highlights request bucketing and FIFO scheduling with a runtime estimator. @SuJinYan123 Decode routing balances both running-request count and KV-cache length. Request count approximates scheduling pressure; accumulated KV length captures memory/state pressure that request count alone misses. The engineering commentary describes prefill/decode scaling signals as aggregate compute load for prefill and running requests for decode. @ZhihuFrontier , @SuJinYan123 Expert-parallel load balancing replicates popular experts. Replicas are periodically redistributed to reduce inter-device waiting caused by routing skew. The trade-off is additional HBM consumption against higher throughput, with diminishing returns as replication increases; the useful replication factor is workload-specific. @ZhihuFrontier , @SuJinYan123 CPU scheduling overlaps accelerator execution. Asynchronous scheduling removes idle gaps between decode steps rather than treating host-side orchestration as free. @ZhihuFrontier One technically notable observation is the apparent absence of tensor parallelism in the disclosed Blackwell configuration. A commentator contrasts this with default vLLM/SGLang recipes and traces the approach back to DeepSeek V3’s H800-era interconnect constraints. That is an interpretation of the configuration—not evidence that tensor parallelism is generally inferior. @nrehiew_ Scale also matters. Su Jin Yan argues that wide expe