Amazon SageMaker adds automated concurrency tests for AI endpoints

Image: Primary

Published Tuesday, September 22, 2026 · 2:17 PM EDT

Amazon SageMaker AI now includes concurrency sweeps in Inference Recommendations, giving operators a managed way to measure how generative-AI endpoints behave as simultaneous requests increase. The benchmark records throughput and latency at each traffic level, exposing the point where throughput stops rising and requests begin to queue. Operators can also use an SLA-based search to find the highest concurrency that remains within specified latency limits. AWS's example found 320 concurrent requests met a 50-second p99 latency target, but results depend on the model, hardware and workload profile tested.

Sources

AWS Machine Learning Blog

In this story

Amazon

Published by Tech & Business, a media brand covering technology and business. This story was sourced from AWS Machine Learning Blog and reviewed by the T&B editorial agent team.

Back to Newswire

Keep reading

Full wire

AI Infrastructure

Goldman and Apollo discuss selling portions of SpaceX chip financing

Goldman Sachs and Apollo Global Management are in talks with Wall Street investors to begin selling portions of a roughly $40 billion chip financing deal for SpaceX, Bloomberg reported. The discussions concern portions of the financing rather than a completed sale.

AI Infrastructure

Google makes Agent Substrate generally available on GKE

Google announced general availability of Agent Substrate on Google Kubernetes Engine and the runtime's 1.0 release.

Infrastructure AI

OpenAI pledges to avoid NDAs in data center negotiations

OpenAI said it will avoid nondisclosure agreements in data center negotiations and ensure residents do not pay for electricity used by those projects.

AI Infrastructure

Nvidia details AI agents operating across its production GPU clusters

Nvidia says it uses AI agents across its DGX H100, B200 and GB300 production clusters to investigate failures and recommend recovery steps.

Infrastructure

AWS adds file-based log collection to ECS Managed Instances

AWS has added support for application logs in Docker's json-file format on ECS Managed Instances, allowing customers to collect them with their chosen monitoring agent and forward them to any destination. The company says the feature carries no additional cost.

Capital Infrastructure

VÆRIDION raises €30 million and wins selection for €94.5 million EU grant

Munich electric aircraft developer VÆRIDION has raised €30 million in Series A2 financing and been selected for a €94.5 million European Commission Innovation Fund grant, EU-Startups reports. SHIFT Invest and Invest-NL co-led the equity round.