Data centers are moving to redesign their infrastructure to accommodate racks that may consume up to one megawatt, driven by the growth of AI workloads, which have become more continuous and energy-intensive. But this trend does not settle the debate over the best way to scale: should more processing units be packed into extremely high-density racks, or should computing be redistributed across a more dispersed architecture?
Semiconductor Engineering observes that the move to megawatt racks is not limited to increasing the number of accelerators; it imposes fundamental changes in cooling, power delivery, rack design, networking, and three-dimensional integrated-circuit packaging. At the same time, optics, specialized chips, and distributed computing could reduce the need to concentrate all this power in one area.
From a Server Container to an Integrated Computing System
Google, Meta, and Microsoft are working within OCP's Mount Diablo project, built on the Diablo framework, with the goal of establishing standards that support megawatt racks. The collaboration includes elements such as rack mechanical architecture, cooling, fluid management, water flow, and power sources, while each cloud provider retains its own implementation path.
Arif Khan of Cadence points out that the rack is no longer merely a passive structure housing a group of servers. The higher computing density and number of electrical and signal connections, along with the increasing size and power consumption of GPU units, make the rack an integrated design unit that includes liquid cooling, power shelves, busbars, network switches, and system management. This approach appears in rack-scale platforms such as Nvidia DGX.
Google is proposing ±400-volt direct-current power distribution inside the data center, drawing on technologies and expertise developed for the electric-vehicle industry. Raising the distribution voltage helps reduce current, losses, and cable complexity when handling power of up to one megawatt.
At the mechanical level, the Open Rack Wide design, or ORW, expands the capabilities of the Open Compute Project architecture to accommodate larger, heavier AI systems with higher power density. ORW is approximately twice as wide as an ORV3 rack, with additional structural reinforcements. AMD is using this approach in the Helios architecture, based on the ORW specification that Meta submitted to OCP in 2025, while Nvidia places rack-scale DGX systems within more traditional rack dimensions.
Three-Dimensional Packaging and Power Challenges
As density increases, 3D-IC design becomes more important. The article quotes Daniel Wilkinson of Synopsys as saying that major cloud providers are studying the stacking of compute dies, with possibilities including increasing the number of dies on the interposer or stacking two dies together. However, these architectures increase mechanical complexity, manufacturing-yield challenges, and the difficulty of moving data across long distances.
Solutions may differ according to the nature of the processor. Some designs, such as Google's TPU, rely on a large computational array, while others are moving toward a larger number of smaller cores in the manner of a GPU. This results in differences in stacking approaches and the requirements for interconnect addresses and foundational IP.
Rack power density has changed from approximately 4 kilowatts 20 years ago to more than 100 kilowatts today, according to Steven Woo of Rambus. With the transition from 12 volts to 48 volts and possibly to 480 or 800 volts, the same amount of power can be delivered at lower current, reducing the need for thicker copper conductors.
Christian Hoefling of Infineon Technologies warns that peak currents for GPU units could reach 10,000 amperes per unit by the end of this decade, compared with much lower ranges in previous CPU systems. Because power losses are proportional to the square of the current, doubling the current while keeping the connection resistance constant causes a substantial increase in losses. Therefore, he expects gigawatt-scale data centers to move toward high-voltage direct-current power distribution to reduce power-conversion stages and simplify delivery to the racks.
Is the Highest-Density Rack the Only Path?
Not all participants in the debate agree that increasing rack power is the best path to scaling. Steve Roddy of Quadric believes that combining more specialized chips with a return to some forms of distributed computing could reduce power density in data centers. Chips specialized for training or inference can provide higher inference per watt, while token generation can also be distributed across millions of homes and businesses in parallel with centralized models.
Ayar Labs, according to Vishal Chandrasekar, is betting that megawatt racks will not achieve widespread adoption. Instead of keeping GPU units close together and relying on copper, optical interconnects can be used to extend the communication domain within the scale-up range to 500,000 units or more. This allows GPU units to be distributed across an entire row of racks—for example, keeping each rack at 200 kilowatts and optically connecting ten racks—while achieving performance roughly comparable to that of a higher-power rack or racks, without completely redesigning power distribution and floor load-bearing capacity every two years.
Reducing rack density does not mean that increasing the number of accelerators guarantees better performance. Communication efficiency, memory bandwidth, latency, collective-operation efficiency, and interoperability are critical factors in making thousands of GPU units operate as a cohesive system. This highlights the role of open scale-up interconnect standards such as UALink.
Agentic Workloads Put Pressure on Design and Timelines
Long-running agentic AI workloads are pushing designers to abandon assumptions based on average consumption and instead account for intermittent peaks. Sathishkumar Balasubramanian of Siemens EDA says that some of these workloads require assuming operation near peak power at all times, with precise analysis of power and thermal behavior. Irregular use, and sometimes continuous, around-the-clock operation, also requires the entire power and cooling system to be designed for this scenario.
Frequent context switching and the requirements for storing, suspending, and restoring models increase the importance of system speed. At the same time, stacked-die architectures and dense transfers make it more difficult to understand the thermal map and physical effects during verification and implementation.
This coincides with a shrinking market window. Manmeet Walia of Synopsys says that some products have only one year to move into production, while specifications are finalized over a period ranging from 1.5 to 2 years. Developing test chips for new technologies, such as HBM, and co-packaging with memory consumes months of the development cycle. Accelerator programs can cost up to $1 billion, making the use of unverified or uncertified IP a major risk.
Segmenting Solutions Instead of a One-Size-Fits-All Architecture
The article raises the possibility that not all inference workloads will run on a megawatt rack. Medium-sized models may account for the majority of inference workloads, while training operations and larger models may require a different architecture. Cloud service providers may also rely on multiple platforms, including Nvidia or AMD products and customized in-house architectures.
Wilkinson points out that institutions will not necessarily place a three-ton, 500-kilowatt rack inside local technical rooms, opening the way for solutions that operate within the capabilities of local sites and for workloads that are not limited to large language models. The article’s conclusion is that the megawatt rack is not a problem of individual racks, but a system-integration problem extending from the electrical grid to the chip, from the chip to the cooler, and from one accelerator to another.