Artificial intelligence

How did LinkedIn develop ultra-fast AI classification systems? Strategies for model pruning and inference optimization

LinkedIn revealed its technical methodology for running large language models (LLMs) in search and recommendation systems, focusing on pruning models from 7 billion to 400 million parameters, and compressing context by 10x to ensure instant response for billions of users.

2026-11-17
4 min read
9 views
certi.news Editorial Team
How did LinkedIn develop ultra-fast AI classification systems? Strategies for model pruning and inference optimization

LinkedIn faces a massive engineering challenge in delivering accurate search and recommendation results to 1.3 billion members worldwide in fractions of a second. With the shift from keyword-based search to natural language search, relying on large language models (LLMs) has become a necessity, but running these massive models in an environment requiring extremely low latency presents a major technical hurdle. Sundara Raman Ramachandran, a senior engineer on the LLM inference team at LinkedIn, outlined the roadmap the company followed to transform these models from research tools into highly efficient production systems.

Model Pruning: From 7 Billion to 400 Million Parameters

LinkedIn began working with a massive model containing 7 billion parameters (7B), which is excellent at understanding context and regulatory policies, but unsuitable for actual deployment due to its slowness and high cost. The team adopted a "multi-teacher distillation" strategy, where knowledge was transferred from the massive model to two smaller specialized models: one to improve "relevance" and another to increase "engagement."

Afterwards, this knowledge was merged into a small language model (SLM) with a size of 600 million parameters, which then underwent a "pruning" process to remove unnecessary layers and attention heads, bringing the final size down to just 400 million parameters. This process allowed the company to obtain a model small enough to operate at ultra-fast speeds while maintaining the quality of classification results within commercially acceptable limits.

Context Compression: Innovation in Token Funding

In classification systems, a "token" represents the basic currency; the greater the number of tokens, the higher the latency. A traditional job description might exceed 2,000 tokens, and when classifying 100 jobs per user, the load becomes immense. LinkedIn pursued two approaches to compress this context:

  • Summarization: Using a language model to summarize the job description from 2,000 tokens down to just 150 tokens, increasing throughput by 9 times with a loss in quality not exceeding 2%.
  • Embeddings: Compressing the entire job description into a single vector (embedding) instead of text, resulting in an additional 10x jump in throughput compared to summarization, with a slight improvement in quality.

Overcoming Python Bottlenecks and the Inference System

Engineers faced unexpected challenges resulting from the nature of the Python language and the operating system. One of the most prominent challenges was sudden spikes in latency (P99 spikes) due to Python's "garbage collector." This problem was resolved by using the gc.freeze() command, which tells the system to ignore long-lived objects and not check them periodically, leading to stabilized latency.

The team also addressed the issue of Python's "Global Interpreter Lock" (GIL), which prevents true parallelism, by transitioning to a multiprocessing system, where each process runs with its own copy of the model and an independent CUDA stream, allowing for full utilization of GPU capabilities.

What changes practically for engineers?

LinkedIn's experience proves that success in running AI at scale does not depend solely on the power of the model, but on the engineering of the surrounding system. The most important lesson here is that "a better representation is not necessarily the longest one"; context reduction and improved software inference efficiency can provide better results than simply increasing model size. For developers, this means the necessity of focusing on improving simple software operations like batch tokenization and using non-blocking CUDA copies before thinking about hardware upgrades or increasing the number of GPUs.

News source
InfoQ - Architecture Articles
Open original source ↗
c
Author

certi.news Editorial Team

In the same category

You may also like

View all news