A Radical Shift Toward Rust
Respan has announced the launch of an open-source AI Gateway written entirely in Rust, following a two-year development journey that included writing over 100,000 lines of code as wrappers around the LiteLLM library. This decision came after a routine update to the LiteLLM library in January 2026 caused intermittent connection failures, prompting the team to seek a more stable and maintainable alternative.
Plugin-Based Architecture
To avoid the maintenance complexities previously encountered, Respan adopted an architecture similar to the Nginx server, where the gateway core is kept lightweight, while core functions such as routing, model providers, and security controls are moved to independent plugins. This approach allows developers to build their own plugins without needing to modify the core source code.
These plugins are characterized by being extremely lightweight; the company noted that integrating a model provider like Mistral requires only 22 lines of code, making it easier to handle technical differences between model providers, such as how token consumption is reported.
What Changes Practically for Developers?
- Protocol Translation: The gateway can convert between different protocols, allowing applications designed for the OpenAI Realtime API to connect to Gemini Live without client-side code modifications.
- Request Control: Requests pass through plugins sequentially, giving developers the ability to modify the request, make additional calls, or even halt the request before it reaches the model provider.
- Performance and Security: Rust provides greater memory safety, but it imposes new challenges; custom plugins must be compiled with the gateway, meaning that adding a new feature requires rebuilding the gateway's binary.
Challenges and Security Considerations
The adoption of compiled plugins has raised concerns within the developer community regarding supply chain risks, as plugins operate within the gateway process itself, granting them direct access to sensitive request data. The company also emphasized the importance of plugin ordering; placing a cache plugin before a security plugin could lead to bypassing security checks if a cached response exists, which requires high precision in configuration.
This trend comes at a time when other projects, such as LiteLLM itself, are seeking to transition toward Rust, albeit in a gradual manner that focuses on replacing critical paths rather than the comprehensive rebuild adopted by Respan.