Dynamic Resource Allocation, or DRA, in Kubernetes does not eliminate the HAMi project for sharing graphics processing units, but it changes the division of roles between them. After DRA reached general availability in Kubernetes v1.34 and was enabled by default starting with v1.35, Kubernetes became able to natively understand and schedule partial device-resource quota requests. Enforcing those quotas inside the container during CUDA calls, however, is not DRA’s job, and this is where HAMi-core remains essential.
A CNCF article written by Mesut Oezdil provides an analytical reading of the difference between the two stages and explains how HAMi is rebuilding part of its ecosystem on top of DRA instead of abandoning the project. The author points out that the question is not whether one project will replace the other, but which of HAMi’s functions are now covered by Kubernetes’ native capabilities.
Why Did GPU Sharing Need Specialized Solutions?
The Kubernetes Device Plugin interface was essentially capable of counting devices. A traditional request such as nvidia.com/gpu: 1 meant reserving an entire card, without a native language for requesting 8,000 megabytes of a card’s memory or 10% of its computing capacity.
To handle this, HAMi uses extended resources such as nvidia.com/gpumem and nvidia.com/gpucores. However, Kubernetes’ default scheduler treats these values as opaque numbers; it does not know that the memory and computing capacity must come from the same physical card, nor can it determine on its own whether multiple quotas will exceed the capacity of a particular card.
Therefore, the traditional approach relies on a webhook to modify the request, a scheduler extension that filters nodes and selects the device ID, and then records the decision in an annotation. The Device Plugin later reads this decision to inject limits such as CUDA_DEVICE_MEMORY_LIMIT_0=8000m and CUDA_DEVICE_SM_LIMIT=10, along with preloading the libvgpu.so library to enforce the limits.
According to the article, DaoCloud ran more than 10,000 GPUs across more than 10 data centers using this approach. However, its architecture remains tied to a special annotation format and components understood by HAMi, which is the gap DRA was designed to address.
What Does DRA Add?
DRA replaces the device-counting model with a claims-based model, with four main objects in the resource.k8s.io/v1 API:
- ResourceSlice: Published by the device driver, it describes the actual hardware on each node, including the model, memory, and architecture.
- DeviceClass: Defines device classes and their filters using CEL expressions.
- ResourceClaim: Created by the workload owner to request a device according to the class, selectors, and constraints.
- ResourceClaimTemplate: Creates a separate claim for each workload replica.
The scheduler assigns a specific device to the claim before attaching the container, and the result appears in the ResourceClaim status as a structured API object. This gives the decision a native location that kubectl can read, RBAC can protect, and other controllers can build upon, instead of storing it as a text string in an annotation.
However, basic DRA alone is not sufficient for HAMi-style memory sharing. The important extension here is Consumable Capacity, which appeared experimentally in v1.34 behind the DRAConsumableCapacity gate and then became experimental and enabled by default starting with v1.36. This extension allows the driver to declare that a device accepts multiple allocations and allows a claim to request a specific quantity of a named resource on the device, such as memory or computing capacity.
This makes HAMi resource matching almost direct: gpumem becomes a memory-capacity request, and gpucores becomes a computing-capacity request, while determining whether the card still has the requested capacity moves from HAMi’s scheduling extension to the Kubernetes scheduler itself.
Scheduling Does Not Mean Enforcing Limits
The analysis emphasizes that DRA tracks the promises made by the scheduler, but it does not prevent the container from exceeding them during execution. CUDA calls do not know what the ResourceClaim says, and a greedy workload may attempt to consume additional memory at the expense of another container.
HAMi-core handles this function through a C library, libvgpu.so, which intercepts CUDA and NVIDIA Management Library calls and applies limits from user space. According to the example in the article, if two containers each receive 8,000 megabytes, the container that exceeds its quota receives a CUDA out-of-memory error at its limit, while the other container continues operating.
This software protection is not a substitute for hardware partitioning in hostile multi-tenant environments. A workload that bypasses library preloading, uses static linking to the CUDA driver, or takes advantage of settings such as CUDA_DISABLE_CONTROL may evade interception. The author notes that NVIDIA Multi-Instance GPU, or MIG, is more suitable for hardware isolation, while software interception provides precision down to increments of 1 megabyte for memory and 1% for computing, compared with fixed MIG profiles.
How Is HAMi Rebuilding Its Ecosystem on Top of DRA?
The new architecture is distributed across three repositories with different functions:
- k8s-dra-driver: Publishes each GPU’s memory and computing capacity as consumable capacity in ResourceSlices, runs the kubelet plugin, and connects containers through CDI while attaching HAMi-core enforcement.
- HAMi-DRA: An admission-modifying webhook that removes traditional extended resources from requests and creates equivalent ResourceClaims, while preserving annotations for UUID targeting and device type. According to the article, HAMi-DRA v0.2.0 became production-ready with HAMi v2.9, after which the release series moved to v0.2.1.
- HAMi: Documents DRA mode as an installation option starting with release v2.8, also enables the monitoring component by default, and exposes per-container device metrics through Prometheus on port 31995.
One of the benefits of HAMi-DRA is that it leaves scheduling to the scheduler managing the cluster, allowing Volcano, KAI Scheduler, or any other scheduler that understands DRA to be used without adding HAMi-specific integration. However, this decision has a cost: HAMi-DRA does not have its own scheduler and therefore cannot guarantee topology-aware decisions, such as selecting a pair of GPUs connected through NVLink.
Requirements and the Practical Choice
DRA mode requires Kubernetes v1.34 or later with DRAConsumableCapacity enabled. In v1.34 and v1.35, the gate is experimental and disabled by default, which may prevent its use in managed services that do not allow API server settings to be modified. In v1.36, it became experimental and enabled by default. A CDI-supporting runtime is also required, such as containerd or CRI-O with CDI enabled, along with an NVIDIA driver version 440 or later and a suitable DRA driver for the accelerator type.
The article describes NVIDIA’s path as the most mature, with support for Ascend and Enflame available and Hygon DCU documented through k8s-dcu-dra-driver. By contrast, traditional HAMi mode covers more than 12 device families as of v2.9, including Cambricon MLUs, Iluvatar, MetaX, Moore Threads, Kunlunxin, AWS Neuron, and Vastai. Therefore, multi-vendor clusters remain candidates for continuing with the traditional path until DRA driver coverage expands.
The DRA mode and the traditional Device Plugin mode should not be enabled in the same cluster, because the two schedulers will appear to be managing the same capacity without visibility into the other system’s reservations. According to the author’s assessment, the traditional mode is suitable for managed clusters that do not expose feature gates, older versions, and multi-vendor fleets. NVIDIA clusters that control the Kubernetes layer, especially on v1.36, can try DRA mode in a test environment, starting with HAMi-DRA to avoid modifying existing deployment files.
Consumable Capacity remains definitively unstable in Kubernetes, while the Helm chart for k8s-dra-driver is still classified as a work in progress, and vendor coverage on the DRA side remains lower than in the traditional mode. The article’s conclusion is that DRA handles the language of requests and scheduling, while HAMi-core retains execution capabilities inside the container; in other words, their relationship is moving toward integration rather than replacement.