DeepSeek releases inference framework for Nvidia and Huawei hardware

DeepSeek has released DeepInfer under the Apache License 2.0, making the inference framework behind its production services available for offline use. The release supports DeepSeek-V4.1-Flash on Nvidia SM100 GPUs and Huawei Ascend 950 processors. Features include generating draft tokens to accelerate decoding, overlapping CPU scheduling with device execution, and constraining outputs to a schema or grammar. Nvidia users can also enable consistent results across different batch compositions and deterministic sampling. The release does not include all components needed for a complete inference service: operators must supply a system for caching model state and load balancing across replicas, among other service infrastructure.