Context
Every AI feature across the three products ran on the same model: screening CVs against job criteria, rewriting job descriptions, translating interface copy in bulk and one string at a time, tidying up support emails. It had been chosen once, under deadline, and never revisited. Then the provider announced a retirement date, and cheaper models appeared on the cloud account we already paid for.
The question was not which model is best. It was which model is good enough for each job, because the jobs are nothing alike. Translating a button label and judging a candidate's CV should not be priced, or trusted, the same way.
Constraints
- The comparison had to use the exact prompts and settings production uses, or it would be measuring a different product.
- Some tasks have a hard floor. Output that production cannot parse is a failure, however eloquent.
- Quality is partly subjective, and I did not trust any single model, least of all one from the same family as a candidate, to grade its cousins.
- Seven target languages across four scripts, one of them right-to-left, so 'looks fine in English' proves nothing.
- It had to be repeatable, because models change faster than budgets do.
Approach
I built the harness with an AI coding assistant as the pair, and that is the part I find most telling. The assistant wrote most of the code. My job was every decision the code would carry: what counts as a task category, what is a hard failure, how the score is composed, what bar a model must clear, and which model is allowed to judge.
- Fixtures per task, real-shaped and deliberately awkward: a CV written to trigger rejection, a job description full of messy markup, inputs far larger than anything typical.
- Deterministic checks first: parseable output, placeholders and markup preserved, ratings inside the allowed range, the right script for the target language. Any check that would make production throw caps the score for that case, whatever else the model did well.
- Then a blind judge. A model from a different family scores each response on task-specific dimensions, such as grounding in the CV, consistent payroll terminology, or phrasing a native speaker would accept. It never learns which model it is grading.
- Half the composite comes from the checks and half from the judge, with several runs per case so a lucky answer does not count.
- Cost is measured from the tokens actually used, at live prices, per task. Latency is tracked at the tail, not the average.
- The selection rule is written down and applied mechanically: per task, take the cheapest model that clears the quality bar with no hard failures and acceptable tail latency. Otherwise stay on the incumbent.
Outcome
The result is a routing table rather than a winner. Budget models cleared the bar on several tasks at roughly a third to a half of the cost, while the incumbent stayed the right answer where nuance mattered more. Nobody would have guessed that mix from a leaderboard.
More useful than the table is the harness itself. When the next model ships, the comparison is a command and an afternoon, not a project.
What I'd do differently
- Pin the judge before anything else. I lost time to quota limits and fell back down a list of candidates. The fallback turned out well, but that was luck.
- One task's production prompt had to be reconstructed from its ingredients, which left an assumption in the results. Every prompt now belongs where the harness can read it verbatim.
- Start with the awkward fixtures. The clean cases separated nobody; the messy ones did all the work.