Together AI has unveiled a new technological innovation named Together Compiler, an advanced system relying on AI coding agents to take over the task of optimizing GPU code that powers its inference infrastructure. This system aims to automate performance tuning processes that previously took human engineers days or weeks of painstaking manual work.
Full automation under human oversight
The Together Compiler system generates, tests, and optimizes GPU kernels without direct engineering direction for each individual experiment. The company explained that the optimization cycle operates completely autonomously, allowing the system to achieve tangible performance gains in a single night. However, Together AI emphasized that it follows a "human-in-the-loop" approach, where final evaluation and actual production deployment remain contingent on a human engineer reviewing the changes before approval.
The challenge of accelerating AI models
This step comes in response to the major challenge posed by the accelerating pace of new AI model launches. According to company data, 80% of traffic through the OpenRouter platform currently comes from models released within just the past three months. Since every model has different technical requirements, code optimized for a specific model may not run with the same efficiency on another, placing tremendous pressure on engineers to keep up with these changes manually.
A leap in experiment speed and performance
Together AI succeeded in inventing a method to significantly accelerate the testing process; instead of restarting the inference engine every time the coding agent changes the code, Together Compiler loads the new code directly into the engine while it is running. This procedure reduced initialization time from about 467 seconds to just 9 seconds, allowing the system to conduct 252 times more optimization experiments compared to existing systems.
On the practical results front, the company recorded an improvement in inference throughput reaching up to 2.6 times compared to standard published configurations, depending on the model type and hardware used. The system currently serves trillions of tokens daily across the company's infrastructure.
What changes practically for developers?
For developers and companies using Together AI's services, this development does not necessarily mean direct price reductions, but rather obtaining higher performance and efficiency for the same cost. The practical benefits are summarized in the following points:
- Hardware efficiency: Extracting the maximum possible power from available GPUs, reducing resource waste.
- Adaptation speed: The ability to run modern models at the highest possible performance immediately upon release without waiting for long manual optimization cycles.
- Performance reliability: Thanks to conducting hundreds of automated experiments, the system arrives at software solutions that a human engineer might not discover due to time constraints.
Together Compiler represents an important shift in how AI infrastructure is managed, where artificial intelligence is no longer just a final product, but has become the primary tool for optimizing the hardware and software it runs on, closing the technical loop toward self-optimizing systems.