Amazon ECS adds GPU sharing for NVIDIA MPS

Efficient GPU Utilization

Amazon Elastic Container Service (Amazon ECS) has introduced support for GPU sharing using the NVIDIA Multi-Process Service (MPS). This update allows users to run several tasks at the same time on a single NVIDIA GPU. Previously, many workloads might have left significant portions of a GPU's power unused because they only required a fraction of its total capacity.

By enabling this sharing capability, companies can significantly increase how much work they get out of their hardware. This is particularly useful for workloads that are not massive in scale but still require GPU acceleration. It helps organizations maximize their existing infrastructure while lowering the overall cost of running these services.

The primary goal of this feature is to improve efficiency for specific types of AI and machine learning tasks. Instead of dedicating an entire expensive GPU to a single small process, users can now pack multiple processes onto the same instance. This leads to better resource management and more predictable spending on cloud infrastructure.

Ideal Workloads for Sharing

The new GPU sharing feature is designed for tasks that do not need to occupy the full memory or compute power of a high-end GPU. For example, it is well-suited for small-model inference, which involves running smaller AI models to get quick answers. It is also useful for embedding generation, where the system creates numerical representations of data for search or retrieval.

Other specific use cases mentioned by the provider include AV/ADAS simulations and multi-model serving. These types of applications often involve running several different models or simulations simultaneously. Because these tasks are often smaller in scale, they are perfect candidates for sharing the resources of a single hardware unit rather than requiring their own dedicated GPU.

One of the biggest advantages for developers is that they do not have to change their application code to take advantage of this. The sharing happens at the infrastructure level managed by Amazon ECS. This means teams can adopt the new efficiency gains without having to rewrite the core logic of their existing software or models.

How the System Works

When using this feature, users define the specific GPU memory that each container needs. They also have the option to set a maximum share of the GPU compute for each task. Amazon ECS then handles the complex job of placing these tasks on GPUs that have enough available capacity as the services scale or deploy. This automated placement ensures that resources are distributed effectively across the fleet.

The technical backbone of this functionality is the NVIDIA Multi-Process Service, or MPS. This service manages how multiple processes access the GPU hardware. By using MPS, the system can coordinate the execution of different tasks so they don't conflict with one another while still sharing the underlying hardware components.

The service is supported across two different deployment models: Amazon ECS Managed Instances and Amazon ECS on Amazon EC2. This means users can choose the management style that best fits their needs while still benefiting from the shared GPU architecture. Both options provide a path toward higher density and lower costs for containerized GPU workloads.

Reliability and Management

For users who prefer the hands-off experience of Amazon ECS Managed Instances, the service includes built-in monitoring. Amazon ECS tracks the MPS control daemon, which is the software component that enables the sharing on each instance. If this daemon stops working or is flagged as unhealthy, the system takes automatic action to maintain uptime.

In the event of a failure, Amazon ECS will automatically replace the affected instance. It then moves the running tasks to healthy capacity so that the GPU workloads continue to run without manual intervention from the user. This automation is critical for maintaining high availability in production environments where downtime can be costly.

To implement these changes, users can use several standard tools including the AWS Management Console, AWS CLI, and AWS SDKs. They can also use infrastructure-as-code tools like AWS CloudFormation. The process involves creating a capacity provider with NVIDIA GPU instances and then setting the sharing strategy and memory limits in the task definition.

Availability and Next Steps

This new capability is currently available in all AWS commercial regions. It is also available in the AWS GovCloud (US) Regions, making it accessible for government and regulated industries that require specific cloud environments. This broad availability ensures that a wide range of organizations can begin optimizing their GPU costs immediately.

Organizations looking to reduce their infrastructure spend should evaluate their current GPU workloads to see which ones use only a portion of the hardware. By identifying these small-model inferences or simulations, they can begin migrating them to a shared GPU configuration. This shift can lead to a more streamlined and cost-effective cloud architecture.

The documentation provided by the outlet details the specific steps for creating capacity providers and defining task requirements. As more companies move toward multi-model serving and complex simulations, the ability to share hardware will likely become a standard practice for efficient AI deployment. This update represents a step toward making high-performance computing more accessible for diverse applications.