AWS adds hardware GPU partitioning to SageMaker inference endpoints

AWS says Amazon SageMaker Inference now lets customers divide supported GPUs into hardware-isolated partitions, allowing different models to share an accelerator with dedicated compute and GPU memory. The feature supports P4d, P5, G7 and G7e instances in seven AWS Regions. Customers select a partition layout when creating an endpoint and assign models through inference components, which specify each model's resources. Support is limited to real-time inference component endpoints; serverless endpoints and heterogeneous instance pools are excluded. Customers still pay for the full instance. Changing the GPU layout requires a new endpoint and traffic migration, and models sharing an instance still share host resources, including CPU and networking.