Split the work
Planning and formatting a hundred records have different demands. Describe the input, output and cost of an error for each task. Use strong reasoning for difficult choices when your test justifies it. Simple, checkable work is a candidate for a cheaper model. Treat this as a hypothesis, not a universal ranking.
Set the quality threshold first
Define a valid result before seeing model responses. For formatting, check required fields and counts. For a plan, specify constraints and unresolved decisions. A second model can help review, but use a check outside the model when possible.
Compare the same task
Give both models identical input and the same output contract. Record the result, duration, failure type and repair work. Start with a small fixed test set. A cheaper response requiring three repair attempts can cost more than an initially usable response from a more expensive model.
Measure cost per usable result
Include model calls, retries and paid tool usage. Record human correction time separately. Use current provider billing; this guide promises no fixed savings percentage. Choose the cheapest model that passes your threshold and keep failure cases for the next test.
Make fallback explicit
Failover after a provider error differs from escalation after a poor answer. Write separate rules. OpenRouter supports model fallbacks for certain error conditions; that does not replace your own quality review. Do not silently route confidential input to another provider. Test failure handling with test data.
Your worksheet
# Model test card Task: [one recurring task] Fixed input: [file and version] Valid result: [three testable requirements] | Trial | Model | Valid? | Errors | Cost | Duration | | A1 | [model A] | [yes/no] | [specific] | [actual] | [actual] | | A2 | [model A] | [yes/no] | [specific] | [actual] | [actual] | | B1 | [model B] | [yes/no] | [specific] | [actual] | [actual] | | B2 | [model B] | [yes/no] | [specific] | [actual] | [actual] | Cost per valid result = total trial cost / valid results. Zero valid results means unusable output; do not divide by zero. Primary route: [model and reason] Technical-error fallback: [model and error condition] Content uncertainty: [human review]
STARTER
Make it concrete.
Create a routing card for these three tasks. Use only the models and prices I supply. Define acceptance criteria first. Compare the supplied outputs on quality, retries, cost and repair work. Recommend a model per task, with separate rules for provider errors and inadequate output. Mark missing numbers unknown.
Sources and further reading
The AI Operator · M3.B.3, M3.B.4, M3.B.8, M3.G.6