The Routing Problem
When you call api.infyrence.com/v1/chat/completions with model: "openai/gpt-4o", several decisions happen in under 10 milliseconds.
Step 1: Provider Resolution
We resolve the model ID to a list of eligible provider endpoints. For first-party models (GPT-4o, Claude), that is typically one provider. For open-source models (Llama, Mistral), we may have 5–10 hosting providers to choose from.
Step 2: Latency Scoring
Each endpoint gets a real-time latency score based on:
- Rolling P50/P95 from the last 60 seconds
- Current queue depth (from provider status APIs where available)
- Regional proximity to the originating request
Step 3: Health Check Gating
Endpoints that have failed health checks in the last 30 seconds are excluded from routing. We run synthetic health checks every 10 seconds per endpoint.
Step 4: Fallback Chains
If the primary endpoint fails mid-stream, we restart the request on the next eligible endpoint. The client sees seamless streaming with a retry transparent to them.
The Result
99.99% uptime across 200+ endpoints, with median routing overhead under 2ms.
