Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit

Xiaomi researchers detailed engineering optimizations for the MiMo-V2.5 and MiMo-V2.5-Pro inference systems, focusing on realizing the theoretical efficiency of the Hybrid Sliding Window Attention architecture. The models combine Hybrid SWA, sparse Mixture of Experts activation, and multimodal encoders for vision, audio, and video. The company said Hybrid SWA interleaves 10 Full Attention layers with 60 SWA layers using a window size of 128, reducing compute and KVCache storage to roughly one-seventh of Full Attention equivalents. Researchers said theoretical gains required solving storage conflicts where traditional single KV pools forced O(N) allocation for all layers, negating SWA sparsity. They implemented separate KVCache pools for Full Attention and SWA layers, improving capacity efficiency