所有 Inferred 期刊

Inferred 06

A new routing engine

This week - the router. On Wednesday we shipped the biggest change to provider routing since OpenRouter began. Instead of mostly sending traffic to the cheapest provider, the router now weighs speed and reliability alongside price for every request.

亮点

  • Faster first touch: First tokens arrived sooner on 15 of the 20 models we tested, by as much as 42% on DeepSeek V3.1.
  • Fewer rate limits: The new router hit fewer 429s than the old one in 27 of 28 head-to-head blocks.
  • Faster streams: Output tokens per second rose on 14 of the 20 models, by as much as 88% on Qwen3.8 27B.
  • Across the whole experiment, median time to first token fell 16.9% and the share of requests hitting a 429 roughly halved, for 6.0% more cost per request.:

Speedy initial tokens

Each bar is one model. Left of zero, the new router got the first token sooner.
Each bar is one model. Left of zero, the new router got the first token sooner.

Time to first token is of paramount importance not just to the experience of conversing with models but also in the long trains of multi-turn agent workflows. The new router improves that snappy touchpoint across most models. We score every provider for every model separately, so the gains can vary a lot by model. DeepSeek V3.1 got its first token 42% sooner. Qwen3 32B, Qwen3.8 27B, Kimi K2.6 and MiniMax M3 were each 20 to 28% faster.

Five models were flat or slower. Interestingly, these tended to be models where the various providers were already quite close. There was less room to improve across available inference and a provider that is cheaper but slower still saw a solid share of tokens.

Consistently reliable

Requests that hit a 429 before succeeding. Each point is a 3-hour block.
Requests that hit a 429 before succeeding. Each point is a 3-hour block.

To measure the new router’s reliability fairly, we switched back and forth between the two routers every three hours for a full week. The blue line is the new router and the gray line is the old price-first one. Each point is a 3-hour block.

The blue line sits below the gray one in 27 of 28 pairs, across weekdays, the weekend and overnight. The typical old-router block had about 6% of requests hit a 429 before succeeding. The typical new-router block had about 2.5%.

This improved reliability is downstream of an increasingly watchful eye on capacity. When we see a provider start to turn away requests, that provider receives proportionally less traffic until it recovers, no matter where it sat in the original routing order.

Faster streams

Each row is one model: gray is the legacy router, blue the new one.
Each row is one model: gray is the legacy router, blue the new one.

A quick first token is great, but what about the rest of the response? We also measured how quickly each model streamed once it started. This new router saw output tokens per second rise on 14 of the 20 models. Qwen3.8 27B streamed 88% faster, Llama 3.1 70B 81% faster and DeepSeek V4 Pro, the largest model in the experiment, 17% faster.

A few models moved the other way. DeepSeek V3.1 streamed 5% slower, though it had the biggest TTFT gain of any model.

These improvements do sometimes come at a cost. Across the experiment the new router cost 6.0% more per request on average. For traffic where cost matters most, you can add :floor to the model ID or set provider.sort: "price" on the request. You can also make price your default sort in account settings.

Explicit instructions vs letting it ride

Open-weight chat requests by who chose the provider, October 28 to 29, 2026.
Open-weight chat requests by who chose the provider, October 28 to 29, 2026.

A quarter of open-weight requests on OpenRouter still name their providers with provider.order, only or ignore, and another 11% set an explicit sort. Those requests will obviously not benefit from the new router improvements.

Plenty of those pins have good reasons behind them, such as data policies, contracts, or eval parity. But we expect a lot of them were originally set to avoid a slow or flaky provider. That’s now the router’s job! If you pinned for speed or reliability, try dropping the pin on one workload and compare. And on the off chance pinning was in order to maintain cache, don’t worry - once a conversation lands on a provider, it stays there while the cache is warm.

We’re hopeful that this new router provides a foundation for an ever-improving inference market. Providers can now optimize across multiple dimensions of quality rather than being artificially focused on cost alone (or even driven to playing pricing games with list vs effective price). We wanted to incentivize a market where competing providers won traffic by serving better inference, not just cheaper inference.

Onwards,

Peter

Past issues of Inferred are on the web at openrouter.ai/data/newsletter.