When our inference bill passed our payroll, the obvious fix was to route every request to the cheapest model that could answer it. We tried that for a week. Costs fell, and so did the answers.
The wrong objective
Cost-first routing optimises a number your customers never see. The number they do see is whether the answer was right. So we turned the problem around: set the quality bar first, then let cost break the tie.
Setting the bar
We scored 1,800 real conversations with a rubric our support leads wrote, and picked 0.92 because that is where complaints stopped. The router config is four lines.
router:
model: auto
quality: 0.92
fallback: [open-70b, frontier-xl]“We expected a migration. It was a config change.”
What happened to the bill
Costs fell 62% against the single-model baseline in two weeks, while quality stayed above the bar. Most requests never needed the frontier model; the ones that did still got it.
What we would do again
Write the rubric before you pick a model. Version the test set like code. And put cost second, where it belongs.
