LinkedIn faces a massive engineering challenge in delivering accurate search and recommendation results to 1.3 billion members worldwide in fractions of a second. With the shift from keyword-based search to natural language search, relying on large language models (LLMs) has become a necessity, but running these massive models in an environment requiring extremely low latency presents a major technical hurdle. Sundara Raman Ramachandran, a senior engineer on the LLM inference team at LinkedIn, outlined the roadmap the company followed to transform these models from research tools into highly efficient production systems.
Model Pruning: From 7 Billion to 400 Million Parameters
LinkedIn began working with a massive model containing 7 billion parameters (7B), which is excellent at understanding context and regulatory policies, but unsuitable for actual deployment due to its slowness and high cost. The team adopted a "multi-teacher distillation" strategy, where knowledge was transferred from the massive model to two smaller specialized models: one to improve "relevance" and another to increase "engagement."
Afterwards, this knowledge was merged into a small language model (SLM) with a size of 600 million parameters, which then underwent a "pruning" process to remove unnecessary layers and attention heads, bringing the final size down to just 400 million parameters. This process allowed the company to obtain a model small enough to operate at ultra-fast speeds while maintaining the quality of classification results within commercially acceptable limits.
Context Compression: Innovation in Token Funding
In classification systems, a "token" represents the basic currency; the greater the number of tokens, the higher the latency. A traditional job description might exceed 2,000 tokens, and when classifying 100 jobs per user, the load becomes immense. LinkedIn pursued two approaches to compress this context:
- Summarization: Using a language model to summarize the job description from 2,000 tokens down to just 150 tokens, increasing throughput by 9 times with a loss in quality not exceeding 2%.
- Embeddings: Compressing the entire job description into a single vector (embedding) instead of text, resulting in an additional 10x jump in throughput compared to summarization, with a slight improvement in quality.
Overcoming Python Bottlenecks and the Inference System
Engineers faced unexpected challenges resulting from the nature of the Python language and the operating system. One of the most prominent challenges was sudden spikes in latency (P99 spikes) due to Python's "garbage collector." This problem was resolved by using the gc.freeze() command, which tells the system to ignore long-lived objects and not check them periodically, leading to stabilized latency.
The team also addressed the issue of Python's "Global Interpreter Lock" (GIL), which prevents true parallelism, by transitioning to a multiprocessing system, where each process runs with its own copy of the model and an independent CUDA stream, allowing for full utilization of GPU capabilities.
What changes practically for engineers?
LinkedIn's experience proves that success in running AI at scale does not depend solely on the power of the model, but on the engineering of the surrounding system. The most important lesson here is that "a better representation is not necessarily the longest one"; context reduction and improved software inference efficiency can provide better results than simply increasing model size. For developers, this means the necessity of focusing on improving simple software operations like batch tokenization and using non-blocking CUDA copies before thinking about hardware upgrades or increasing the number of GPUs.