Follow the latest coverage, related explainers and connected technology stories.
Data centers are moving toward heterogeneous clusters that combine CPUs, GPUs, NPUs, custom accelerators, and optical interconnects, rather than relying on a single GPU for all tasks. This shift makes software, network management, and power key factors in reducing token costs and improving hardware utilization.
The startup Etched raised $700 million at a valuation of $21 billion after Jane Street tested the first complete artificial intelligence system shipped by the company and purchased the system to operate it in its data center. The funding reflects a bet on the company’s designs to accelerate AI model inference and reduce its cost.
Sharad Chol of Expedera analyzes how the transition of edge devices from traditional vision networks to LLM and VLM models is shifting the nature of the bottleneck from computational capacity to memory traffic. He outlines the role of packet-based processing in reducing external data transfers and improving model execution inside vehicles and embedded devices.