The challenge of variable lengths in edge devices
When designing Large Language Models (LLMs) to run on edge devices, developers face a fundamental challenge regarding the varying lengths of "prompts" received by the device. While a prompt might be limited to a simple voice command consisting of a few words, other operations may require the analysis of long documents containing thousands of tokens. This variance imposes significant pressure on how computational resources are managed in devices with limited capabilities.
The problem of "padding" and computational energy waste
Most current inference stacks rely on processing graphs with fixed lengths. For example, if the system is set to process 1024 tokens, it will process 1024 positions every time the model is run, even if the actual prompt consists of only 50 tokens. In this case, the system is forced to process the remaining 974 tokens as useless "padding" operations, leading to significant waste in time and computational energy.
To overcome this problem using traditional methods, developers are forced to build multiple binary files, with each length range having its own specific file, which significantly increases the complexity of qualification and software management.
The Quadric Chimera GPNPU solution
The General Purpose Neural Processing Unit (GPNPU) from Quadric, known as Chimera, offers a completely different approach. Instead of relying on fixed lengths, this unit ensures the ability to use one graph to serve all possible lengths up to the maximum capacity configured.
This innovation eliminates the need to build multiple binary files for each length, reducing complexity in the software stack and ensuring that computational resources are utilized with maximum efficiency without wasting processing cycles on unnecessary padding operations.
What does this mean for developers and manufacturers?
This development represents a shift in how AI performance is optimized on edge devices, providing the following advantages:
- Energy efficiency: Reducing unnecessary computational operations means lower power consumption, which is vital for battery-operated devices.
- Software simplification: Reducing the number of binary files that must be built and tested accelerates the product development cycle.
- Flexibility: The ability to handle diverse inputs (from short voice commands to long texts) using the same software infrastructure.