OpenAI has revealed details about its development of a new Git backend system called Pando, specifically designed to support the company's massive monorepo and cope with the rapid growth in code commits generated by its developers and AI agents. The company stated that the number of quarterly commits has surged by over 2,400% compared to the first quarter of 2024.
The new system has reduced push latency by approximately five times, while the system currently processes nearly one petabyte of Git data daily. During the development process, which took 10 weeks using the Rust language with a large portion of the code programmed by Codex agents, OpenAI decided to maintain compatibility with the standard Git protocol rather than building an entirely new client system and software architecture from scratch like other companies such as Anthropic.
How Does the New Pando System Work?
Pando overcomes traditional Git server bottlenecks by dropping two fundamental assumptions that traditional servers treat as mandatory; it does not maintain a consistent state for all repository branches simultaneously, and it does not generate custom packfiles for each individual client during fetches.
- Eliminating Global Coordination: The system handles each branch independently, preventing pushes on unrelated branches from waiting due to updates on other branches.
- Immediate Verification and Caching: The system verifies packfiles as they are streamed to cloud storage without locally unpacking them, while a caching service called Express handles serving the packages over the network with a success rate exceeding 99%.
- Backend Compaction: Backend servers gradually merge small packages into larger ones over time in a manner similar to log-structured databases.
Contributions and Open Limitations
Alongside the internal Pando system, OpenAI had to fix several flaws in the native Git client related to concurrent package downloading and resuming interrupted transfers, submitting some of these fixes and other additions like parallel indexing to the open-source project, while the Pando backend system remains closed-source for now.