Microsoft details 96-chip Maia 300 racks and expanded inference network

Microsoft detailed a Maia 300 design that doubles accelerator density to 96 chips in a 167-kW rack and connects up to 12,288 accelerators through one network. The company says the full rack delivers up to 7 times Maia 200's token throughput and more than 2.5 times its performance per watt. Each accelerator has 288 GB of high-bandwidth memory with 18 TB/s of bandwidth. Software places data near the compute that uses it, while electrical links connect chips within racks and optical links connect racks. The network lets inference jobs span racks and move stored context directly between workers. Using spare capacity in another rack improves availability but adds communication latency.