Artificial intelligence data center architecture is moving toward heterogeneous computing clusters that combine central processing units (CPUs), graphics processing units (GPUs), neural processing units (NPUs), and custom accelerators, alongside high-bandwidth memory and copper and optical interconnect technologies. This shift reflects the differing nature of model training and inference tasks, particularly as agentic AI applications grow and require a mix of parallel computation, state management, decision-making, and tool-call execution.
This was discussed in a panel hosted by Semiconductor Engineering with Satadal Bhattacharjee of Arm, Ashish Darbari of Axiomise, Moshiko Emmer of Cadence, Sharad Chole of Expedera, Cameron Brunner of Siemens EDA, and Sumit Vishwakarma of Synopsys. The article is the third and final installment in a discussion series about changes in AI data center architecture.
From a Powerful Server to a Distributed System
Multi-processor clusters change the nature of the engineering problem itself. Instead of designing a single server that balances processing, memory, and input/output, performance becomes tied to workload partitioning, communication patterns, and the interconnect network’s ability to transfer data and perform synchronization operations efficiently.
The participants explained that different parallelism models, such as Tensor parallel, data parallel, context parallel, and pipeline parallel, require different forms of network topology. Pipeline parallelism connects successive stages, while tensor parallelism requires broad aggregation operations followed by broadcasting the results again. Therefore, selecting the interconnect network becomes part of defining the cluster, rather than a component separate from it.
Memory becomes increasingly important as models grow. In addition to the model’s fixed parameters, there is changing context affected by the number of requests being processed and the volume of information the system retains. This makes the cost of HBM memory and how it is used influential factors in GPU efficiency, particularly in data centers seeking to increase the number of requests processed per second.
Disaggregating Inference Across Specialized Clusters
Satadal Bhattacharjee believes that one important trend is disaggregating the inference pipeline into separate stages. The process begins with the prefill stage, which processes the prompt and determines what is being requested; this is a computation-intensive stage. It is followed by the decode stage, which generates the response, and then by the agent task-execution stage, such as calling tools or carrying out a specific action.
In this model, one hardware and software cluster can be dedicated to the prefill stage, another to the decode stage, and a computing cluster to executing agent tasks, after which these clusters are connected, often through Ethernet, according to the discussion. This makes it possible to use each type of hardware for the task it suits instead of operating a single GPU to perform all stages.
Bhattacharjee pointed to Nvidia’s announcement that it would use the Groq 3 LPU in a prefill cluster as an example of how inference may require more than one type of processor. He also mentioned that DigitalOcean had announced deploying a five-layer inference architecture optimized for using heterogeneous clusters containing hardware from AMD and Nvidia, with the possibility of adding new hardware later.
Software Is the Coordination Layer
It is not enough to combine CPUs, GPUs, NPUs, and accelerators in a single system. Software must understand the characteristics of each component, decide where each stage should run, and coordinate data transfers, scheduling, waiting, and access points to application programming interfaces. CPUs can handle state management, instruction routing, and tool calls, while accelerators handle computationally intensive operations, including the matrix multiplications associated with inference and reasoning.
Virtual machines are used to abstract clusters at a higher level, but the participants described container-based environments, including Docker, as the prevailing method for deploying components and GPU driver toolchains reproducibly. Above the container layer are specialized components for execution, coordination, scheduling, queue management, model deployment, and serving application programming interfaces.
The participants considered ownership of an integrated software ecosystem one of Nvidia’s strengths, after the company invested in software for more than 20 years. However, the move to hardware from multiple companies requires a coordination layer capable of managing this diversity. Companies such as Gimlet Labs and Together AI were mentioned as working to provide this layer, including optimizing the operation of prefill, decode, and execution clusters.
Interconnects and Power Determine Scalability
Interconnect options between clusters include Ethernet, InfiniBand, and Slingshot, a high-speed networking technology based on Ethernet from Hewlett Packard Enterprise. According to the discussion, Ethernet provides a broader standard, while proprietary technologies may offer better performance in certain cases at the cost of being tied to a specific vendor ecosystem.
Optical interconnects are also attracting increasing attention. Google uses TPU units with optical links, while technologies for optics integrated into the package, or co-packaged optics, are being researched. Photonics are attractive because they reduce resistance and power dissipation compared with electrical links, which is important in clusters already facing challenges related to heat and cooling costs.
At the package level, integrating multiple chips, HBM, and different components increases the complexity of thermal, mechanical, and electrical effects. Heat, warpage, electromigration issues, or signal integrity problems may affect the performance of the node and the entire cluster. Therefore, there is less need for one universal interconnect technology than for interconnect layers compatible with every level, from die-to-die connections inside the package to the cluster network.
Why Does This Trend Matter?
The conclusion presented by Bhattacharjee is that power is the primary problem, followed by token efficiency. Heterogeneous clusters offer a path to reducing the cost per token by matching hardware to the nature of each stage, but achieving this requires mature coordination software and clear measurements of data location, bandwidth, thermal state, and each engine’s specialization. Because this ecosystem generally relies on components from multiple companies, building a system that operates efficiently at scale remains an open engineering challenge, not merely a matter of replacing one GPU with another accelerator.