Microsoft has added four experimental types to the Microsoft.Extensions.AI library for managing AI request routing and failover among models or service providers. All of these types implement the IChatClient interface, allowing them to be used within the library’s existing architecture to address cost, availability, and response-time constraints.
Routing and Failover Tools
RoutingChatClient is the base class that selects an appropriate client for each request and then forwards the call to it. It can be created using a simple callback function or extended by overriding SelectClientAsync to implement more complex routing policies or policies that depend on application state.
SemanticRoutingChatClient routes requests based on their meaning. The developer provides the library with examples of phrases associated with each client. The user’s latest message is then converted into an embedding and compared with the examples to select the client with the highest similarity above the specified threshold. If no match exceeds the threshold, the default client is used.
Embeddings for route examples are created on demand and cached. Settings such as scoreThreshold, topK, and scoreAggregation control the minimum similarity, the number of examples used to aggregate the score, and whether the average or sum is used. By default, the type also disposes of the clients and embedding generator when it is disposed, and this can be disabled through leaveOpen.
Failover and Attempt Monitoring
FailoverChatClient extends routing capabilities by adding a retry loop. If the selected client fails before any streaming output reaches the caller, the type calls SelectClientAsync again to select another client. Once output transmission has begun, however, the failure is considered final, and the system does not recover in the middle of the stream.
The class provides OnRoutingUpdateAsync for monitoring every attempt, whether it ends in success, failure, or abandonment. The attempt record includes the client that was called, the execution duration, the exception if one occurred, whether the response completed or streaming output reached the caller, and, when applicable, the time until the first update. This data can be used to evaluate provider performance or build policies such as circuit breaking and ordering clients by response time.
OrderedFailoverChatClient provides a ready-made implementation of this mechanism. It accepts an ordered list of clients and tries them sequentially. If all options fail, the last exception is rethrown. MaximumAttemptsPerRequest determines the maximum number of calls for a request, and selection of a new client stops when the request cancellation token is canceled.
Considerations for Building Routing Policies
The post warns against rerouting on every turn of a conversation without considering the nature of the conversation. Reasoning models may depend on encrypted content or continuation tokens associated with a specific provider, and switching to another model or provider may lose the benefit of prompt caching and incur the cost of computing it again.
For multi-turn conversations, the post suggests pinning the route using a session identifier owned by the service itself rather than relying on the provider’s ConversationId. The selected route can be stored in session state or in IDistributedCache, and the route should be pinned only after the response completes successfully.
Other policies that can be built include routing based on response time, health, cost, capabilities, and geographic region, as well as composing multiple routers because each one operates as an IChatClient. However, these tools do not implement sequential routing based on the quality of a successful response, running multiple clients and merging their results, or racing clients to choose the first response; these scenarios require a client that performs more than one call.
Availability and Experimental Status
RoutingChatClient, RoutingContext, FailoverChatClient, FailoverChatClientAttempt, OrderedFailoverChatClient, and SemanticRoutingChatClient became available in version 10.9.0 of Microsoft.Extensions.AI. All of them are marked as experimental using the diagnostic identifier MEAI001. The package can be added with the command dotnet add package Microsoft.Extensions.AI.