An “AI factory” is not a single model or an individual Kubernetes cluster, but a shared fleet of GPU units used by concurrent teams for different purposes: fine-tuning, inference serving, and evaluation. An article published on the CNCF blog argues that the real enterprise challenge is no longer merely training the model, but giving each team secure, isolated access to the same hardware while keeping GPU utilization high and costs measurable.
The piece was written by Hrittik Roy, a CNCF Ambassador and Platform Advocate at vCluster, and offers a practical look at how to assemble this environment on top of Kubernetes. The article’s central idea is that Kubernetes provides a mature foundation for containers, RBAC, autoscaling, and policies, but requires an additional ecosystem to handle accelerators and isolation among tenants on the same nodes.
The Bottleneck Is Utilization, Not Inference Speed Alone
GPUs represent the largest capital expenditure in AI infrastructure, making the utilization rate a more important economic metric than achieving peak speed in a single run. The article presents two main challenges: the resource allocation model and the isolation model.
In the traditional device plugin model, a workload may request, for example, nvidia.com/gpu: 1 and reserve an entire unit even if it uses only ten percent of it. Dynamic Resource Allocation, or DRA, which became generally available in Kubernetes 1.34, allows the scheduler to treat accelerators as devices with properties, memory, and topology. However, it does not automatically divide a GPU into shares; density comes from the device layer, such as HAMi, a CNCF Incubating project that enforces software limits on memory and compute at the container level and supports multiple accelerator vendors.
By contrast, assigning separate hardware to each team can provide strong isolation but leave a large portion of capacity unused. The article therefore distinguishes between assigning an entire unit when trust boundaries are strict and partitioning the unit within a single trust domain to increase density. NVIDIA MIG provides hardware-level memory and fault isolation, but the article points out that using it as a boundary between hostile tenants remains debated; consequently, assigning a full GPU remains the conservative option in low-trust situations.
The Factory Layers, from Bare Metal to Workload
The system begins with provisioning the raw hardware. Nodes are discovered, GPUs, ECC memory status, and network-card identities are inspected, and an operating-system image containing the GPU driver and CUDA and NCCL libraries is installed. Appropriate BIOS settings are then applied, and the nodes undergo stress tests and NCCL tests to verify that GPU connectivity operates at full bandwidth before the result is recorded in a source of truth such as NetBox. This lifecycle can be built through a vendor-specific hardware manager or by using open tools such as Metal3 with Ironic or vMetal.
After allocation, tools such as KAI Scheduler and Volcano handle batch- and topology-aware scheduling, while Kueue manages waiting, admission, and quotas. At the workload layer, vLLM can be used for inference engines and KServe for providing standard endpoints and autoscaling, along with NVIDIA Dynamo and llm-d for disaggregated inference in larger environments. Gateway API provides routing, while LiteLLM adds an OpenAI-compatible gateway.
The platform is not limited to containers. The article explains that training environments may use Slurm through Slinky from SchedMD, which represents Slurm services as custom resources and integrates them with GPU Operator and DRA. KubeVirt can also run virtual machines as Kubernetes workloads, allowing virtual machines and containers to be managed from the same fleet with unified permissions and quotas.
Isolation Is Not Just Namespaces
The article divides the isolation problem into two levels. At the control-plane level, the tenant-cluster pattern gives each team a virtual Kubernetes interface that includes an API server, dedicated resources, and independent admission rules and RBAC, while running it as a workload on top of a single underlying cluster. vCluster provides an example of this, with the ability to use familiar tools such as kubectl, Helm, and Argo CD without proprietary extensions.
The data plane, meanwhile, requires isolation of networks, storage, quotas, and the runtime environment. Cilium can be used for the CNI and policies, and Multus and SR-IOV for the fast path, with InfiniBand or RoCEv2 for moving GPU traffic between nodes. Separate VPCs can also be used through VXLAN and EVPN, or partitioning keys in InfiniBand, while DPUs such as NVIDIA BlueField or AMD Pensando offload some isolation and encryption functions from the host CPU. The article emphasizes that the standard for a true cloud is hardware-enforced isolation when necessary, rather than relying on namespaces alone.
What Turns the Fleet into a Cloud Service?
Editorial reading: The practical value of this argument is that it shifts the discussion from “Which model is faster?” to the question of operating the infrastructure itself. The platform does not become a cloud service merely by combining GPU units; it does so when a tenant can create and delete clusters through an API, Terraform, or GitOps, and when resources are defined declaratively and managed by Flux or Argo CD, with OIDC identity and RBAC permissions.
The service also needs clear measurement and billing. The article suggests using DCGM data to calculate GPU seconds and then allocating them to tenants through OpenCost. Monitoring and reliability are likewise part of the product, not an operational add-on: DCGM monitors degradation, and Node Problem Detector converts failure signals into node conditions, while the remediation cycle isolates the suspected node and drains it before new workloads are scheduled. The security layer includes tools such as Keycloak through OIDC, OpenBao with External Secrets Operator, Kyverno or OPA for controls, and Falco and Trivy for runtime and supply-chain security.
The hardest test remains moving from a demonstration to large-scale production. An example that runs two teams and two models on one partitioned GPU does not by itself prove that the design is valid across hundreds of nodes and multiple data centers. The article points to the role of validation tools such as NVIDIA AI Cluster Runtime and the Kubernetes AI Conformance program, introduced with release 1.35, but leaves open operational questions about blast radius, tenant-isolation boundaries, and the choice between the NVIDIA DSX OS stack and assembling open-source layers.
The conclusion is that the success of an “AI factory” depends on combining density, isolation, and billing while accounting for hardware topology: the locations of GPU units relative to NVLink or NVSwitch, their connection to InfiniBand or RoCE networks, and the positions of the GPU, NIC, and CPU on NUMA nodes. Scheduling a workload without this information can cause collective communication operations to slow down at the slowest link, even if the platform appears sound at the software level.