AWS makes GPU-aware inference routing available on SageMaker HyperPod

Published Thursday, September 24, 2026 · 8:07 PM EDT

AWS says its SageMaker HyperPod Inference Gateway is available for routing model requests within a cluster in supported regions. It installs as an EKS managed add-on on existing HyperPod infrastructure and works with OpenAI-compatible model servers without application code changes. The gateway reads the requested model and selects a GPU server using signals including queue depth and cache use. AWS says this approach cuts first-token latency by up to 82% and reduces p99 first-token latency by 97-98% in mixed-hardware and burst-traffic scenarios. Routing across clusters and regions is still planned.

Sources

Recent Announcements

In this story

OpenAI Amazon

Published by Tech & Business, a media brand covering technology and business. This story was sourced from Recent Announcements and reviewed by the T&B editorial agent team.

Back to Newswire

Keep reading

Full wire

Infrastructure Products

Two Chinese chip suppliers raise combined $301 million in Hong Kong IPOs

Chip suppliers Suzhou Denglin Technology Co. and Jiangsu Xinde Semiconductor Technology Co. have raised a combined $301 million in initial public offerings, Bloomberg reports.

Security Infrastructure

Researchers report active attacks exploiting Linux SSH vulnerability

Attackers are actively exploiting a vulnerability called sshole to gain unauthenticated access to privileged accounts on some Linux systems, researchers report.

Capital

Altimeter targets $2.5 billion fund for late-stage companies

Altimeter Capital Management is targeting a $2.5 billion venture capital fund for late-stage, pre-IPO companies, Bloomberg reported, citing securities filings and a person familiar with the matter.

Products Infrastructure

Rocket Lab reports $3.57 billion backlog as quarterly revenue rises 72.7%

Rocket Lab reported a $3.57 billion order backlog alongside third-quarter revenue of $267.8 million, up 72.7% from a year earlier.

Security

Vanderbilt tells users to shut down devices during cyberattack

Vanderbilt University confirmed an active cybersecurity incident Monday and told students, faculty and staff to shut down university devices and disconnect from university Wi-Fi.

Infrastructure

Banks and private lenders pull back from new AI data center financing

Commercial banks and private credit lenders have begun pulling back from financing new AI data centers, Business Day reported. Credit committees are stalling deals even for facilities backed by Alphabet and Anthropic as secondary debt markets reprice risk.