In most technical discussions about artificial intelligence, the focus is usually on compute capabilities and the speed of accelerators, which is understandable given the ease of measuring performance through traditional benchmarks. However, the emergence of what is known as "Physical AI" imposes a new reality; where the task is not limited to processing data, but rather how to coordinate work between processing, memory, and data movement under real and complex operational constraints.
Why is processing power alone not enough?
Systems operating in the real world—such as humanoid robots or drones—rely on the ability to see, interpret, make decisions, and then execute in real-time. In these environments, data movement becomes the biggest bottleneck; if the correct data does not reach the appropriate engine at the right time, processing units (such as NPU or CPU) will experience stalls, leading to increased power consumption and the accumulation of latency that could be catastrophic in sensitive systems.
Designing these systems requires a delicate balance between on-chip memory (On-chip SRAM/Cache) and larger capacity external memory such as LPDDR or GDDR. The goal is not to eliminate access to external memory, but to ensure that the most frequently used and time-sensitive data remains very close to the processing units to reduce power consumption and ensure flow continuity.
Real-world applications impose different constraints
Architectural memory requirements vary based on the nature of the device and its application, which illustrates the complexity of the design:
- Humanoids: Face the challenge of balancing limited power consumption with the need to process massive streams of data from cameras, sensors, and motion control. Any delay in the arrival of perception data could lead to failure in making balance or movement decisions.
- Drones: The concept of "Bandwidth per watt" emerges here as a critical metric, as any excessive power consumption in data transfer directly reduces operating time.
- Autonomous Vehicles: Represent the greatest challenge due to the need for sensor fusion from radar, Lidar, and cameras, which creates fierce competition for memory access to ensure immediate and predictive response.
Five pillars for designing an effective memory architecture
To achieve stable performance in Physical AI systems, engineers must make strategic decisions in five key areas:
- Data Locality: Determining which data should remain close to the processor to reduce latency, and which can be stored externally.
- Traffic Contention: Managing the competition between sensor streams, operating software, and safety functions when accessing memory.
- External Memory Suitability: Choosing between LPDDR (ideal for power efficiency and capacity) or GDDR (ideal for high bandwidth for video streams).
- Buffering Strategy: Determining the amount of local storage needed to reuse data and avoid unnecessary off-chip transfers.
- Latency Predictability: Ensuring that latency remains within safe limits even when running multiple tasks simultaneously.
In conclusion, Physical AI represents a true test of system integration; where processing performance cannot be separated from the efficiency of memory engineering, making data architecture design a focus no less important than the design of the processor itself.