All posts
Engineering#Routing#Evals#Open Source
Routing without regret: a quality-aware fallback design
The hardest part of a router isn't picking the cheapest model — it's deciding when to escalate. We open-sourced the eval harness we use internally.
DO
Daniel Okonkwo
Principal Engineer, SpendTensor
Jun 4, 2026 11 min read
Routing
Every router is a bet that the cheap model is good enough. The interesting engineering is the loss function when you're wrong.
Our design has three layers: a cheap classifier picks a candidate model, a streaming quality probe scores the first 60 tokens against a calibrated rubric, and a fallback path swaps providers mid-stream if the score drops below threshold. End-to-end overhead is under 5ms p99.
We're publishing the eval harness under Apache 2.0 this week. It includes calibration data for 9 model families and a replay tool to A/B routing policies against historical traffic.