Terms

KV Cache

Follow the latest coverage, related explainers and connected technology stories.

Latest coverage

CN
Why Are Open-Weight AI Models Reshaping Hardware Design?

Why Are Open-Weight AI Models Reshaping Hardware Design?

Open-weight AI models give organizations greater flexibility for local deployment and reduce reliance on service providers, but they do not automatically mean that training data, code, or usage rights are available. Semiconductor Engineering’s analysis confirms that their impact extends to memory design, interconnects, security, and chip-design-tool workflows.

CN
High Bandwidth Flash Memory Could Reduce Large Language Model Serving Time by up to 87%

High Bandwidth Flash Memory Could Reduce Large Language Model Serving Time by up to 87%

A paper from UC Berkeley and FuriosaAI examines the use of High Bandwidth Flash to expand the memory of AI accelerators when running large language models. The simulations indicate a reduction in completion time of up to 87%, with potential improvements in energy consumption and write endurance through cache-aware scheduling.

CN
Microsoft: The Measure of Progress in AI Should Be “Useful Yield,” Not Infrastructure Scale

Microsoft: The Measure of Progress in AI Should Be “Useful Yield,” Not Infrastructure Scale

Rani Borkar of Microsoft believes that the success of AI infrastructure should not be measured by the number of chips or tokens processed, but by the amount of useful intelligence produced from each unit of energy, memory, and investment. She calls for an integrated design extending from data centers and chips to models, software, and agents.

CN
Why NPUs Need Redesigning as Edge AI Shifts to LLMs

Why NPUs Need Redesigning as Edge AI Shifts to LLMs

Sharad Chol of Expedera analyzes how the transition of edge devices from traditional vision networks to LLM and VLM models is shifting the nature of the bottleneck from computational capacity to memory traffic. He outlines the role of packet-based processing in reducing external data transfers and improving model execution inside vehicles and embedded devices.