Terms

AI Inference: AI Inference

AI inference refers to running trained AI models to produce results in response to requests. This use appears in cloud infrastructure services, enterprise servers, data centers, and chips and systems designed to accelerate output processing.

AI inference is the stage in which a trained AI model runs to process a request and produce outputs. It differs from model training, which focuses on building the model’s capabilities using data and computing resources.

Where is inference used?

  • Infrastructure services: Companies such as Modal Labs provide resources for running models and compute-intensive workloads without directly managing servers.
  • Specialized chips and systems: Etched focuses on hardware systems intended to accelerate inference and reduce its cost and latency compared with the alternatives cited by the company.
  • Data centers and GPU units: Crusoe sells computing capacity for running AI models, in addition to leasing space and GPU units.
  • Enterprise servers: Sharp began accepting orders for servers aimed at businesses, government entities, and data center operators. The announced use cases include training and inference.

Why does the process require specialized infrastructure?

Inference relies on computing resources such as graphics processing units, servers, memory, storage, networking, power, and cooling. The sources explain that providers of these services face rising demand at the same time as high costs associated with purchasing or leasing computing capacity, which may pressure profit margins.

Inference in products and infrastructure

Companies in this field compete across multiple levels: designing custom chips, building customizable servers, providing cloud platforms, and developing data centers capable of running large numbers of GPU units. AI inference is therefore not limited to the model itself; it requires a complete operational ecosystem.

What should be evaluated?

  • Response time: How quickly outputs are produced after a request is submitted.
  • Cost: The cost of providing computing capacity, power, cooling, and operations.
  • Scalability: The infrastructure’s ability to serve larger numbers of requests or models.
  • Reliability: The stability of servers, platforms, and networks during operation.

Limits of the available information

The available sources do not include a standardized comparison of all inference providers in terms of pricing, performance, or market share. Some of the figures mentioned also relate to announced targets, plans, or funding, and do not necessarily represent achieved operating results.