The design of neural AI processing units (NPUs) is changing as edge devices move beyond relying solely on vision networks to running large language models (LLMs), vision-language models (VLMs), and generative AI features alongside traditional perception networks. Sharad Chol, chief scientist and co-founder of Expedera, believes this shift moves the center of the problem from maximizing computation to managing data and memory traffic, particularly in phones, vehicles, and embedded gateways.
The article was published as a sponsored blog post in Semiconductor Engineering. Therefore, the figures and results related to the Origin Evolution architecture are presented as results and experiments cited by Expedera, rather than as independent tests conducted by the publisher.
From a Compute Bottleneck to a Memory Bottleneck
In vision workloads based on CNNs, such as YOLO detectors, MobileNet classifiers, and image-segmentation networks, the primary constraint was historically computational capacity. Data and weights were reused extensively, and execution during inference remained almost stateless, leading accelerator designs to focus on increasing the number of multiply-accumulate (MAC) operations within area and power limits.
Expedera says that scheduling work in packets rather than in large layers improved MAC utilization and reduced activation traffic to external DDR memory. However, LLMs and VLMs change the nature of the workload. The Prefill phase may remain computationally intensive, while the Decode phase quickly becomes constrained by access to the key-value (KV) cache, the size of the parameters and their access patterns, along with lower effective reuse.
According to the analysis, NPUs designed primarily for parallel, stateless CNN processing cannot be evaluated sufficiently using the nominal TOPS figure when transformer decoding operations are limited by memory bandwidth and KV-cache size.
Packets in Transformer Workloads
Expedera is exploring the extension of packet-based processing from CNNs to transformer blocks and KV-cache access, designing hardware and software together according to the behavior of LLM and VLM models rather than treating them as simply another model.
According to the author's description, the architecture consists of separate processing blocks for feed-forward operations, attention, and vector operations. Packets are directed among them according to the network structure and the current execution phase, whether Prefill or Decode. The company says its engine can scale to 128 teraflops in a single core and to the petaflop level when using multicore configurations, while keeping memory behavior at the center of the design.
According to Expedera, the architecture maintains models' operation as they were trained, without retraining or reduced precision. The company also says that, compared with alternative approaches, packet-based processing reduced external memory traffic by more than 75% when running Llama 3.2 1B and Qwen2 1.5B. It also cites results it describes as thousands of effective teraflops and tens of tokens per second per square millimeter of silicon in memory-constrained scenarios.
What Does This Mean for Edge Devices and Vehicles?
Running inference locally can reduce end-to-end latency while keeping cabin or device data outside the cloud. This is important in vehicles and mobile devices, where user-experience requirements intersect with privacy constraints and regulatory requirements, according to the author.
By contrast, streaming the KV cache to DDR or HBM at every decoding step imposes an energy and cost burden, particularly in edge nodes and in-vehicle systems. Expedera believes that reducing external transfers and increasing local reuse could enable more tokens per second within a defined power budget.
Automotive systems-on-chip also need to run perception, driver monitoring, infotainment, and generative features simultaneously. From the author's perspective, an NPU family that handles CNN and LLM workloads together as core workloads could simplify platform design and reduce the need for separate accelerators, rather than adding LLM support to a core originally optimized for CNNs.
The article notes that Origin Evolution received the “Best Edge AI Processor IP” award as part of the 2026 Edge AI and Vision Product of the Year Awards, with the author linking the award to the importance of addressing memory and power bottlenecks rather than merely increasing peak performance metrics.
Modeling the Prefill and Decode Phases
Chol believes that the engineering value lies not in the packet concept alone, but in modeling and optimizing the Prefill and Decode phases separately. The former requires high computational throughput and partially resembles CNN behavior while handling larger models, whereas the latter is dominated by KV-cache reads, memory streaming, and smaller computations per step.
According to the article, packet flows and separate blocks make it possible to build performance and bandwidth models that distinguish between the two phases and to study memory-sequencing and streaming-interface configurations, such as DRAM or HBM bandwidth and SRAM size and partitioning, according to actual KV-cache access patterns. They can also be used to analyze worst-case latency and the impact of bandwidth in safety-sensitive automotive workloads when LLM or VLM features are connected to the human-machine interface or driver monitoring.
Software Compatibility and the Role of EDA Tools
Expedera says that the Origin Evolution package accepts models from HuggingFace, Llama.cpp, TVM, and others, and supports integer and fractional precision, mixed modes, layer fusion or partitioning, as well as centralized control of multiple cores at the chip or multi-die level.
From the author's perspective, packets do not represent a new programming model for model users. Models are converted within the compiler pipeline and runtime environment, with the ability to place trained models on the hardware without retraining or reduced precision. This could reduce the need for custom rewriting for each piece of hardware and allow EDA and verification teams to test performance against reference applications such as Llama 3 and Qwen 2.
The article concludes that the next question is not limited to whether packets work with CNNs, but concerns the extent to which they simplify modeling, scheduling, and verification across heterogeneous AI workloads. As generative features move into vehicles and edge devices, these factors may determine which accelerator architectures move into production and which remain in standards demonstrations.