A cheaper model can leave you with a more expensive task. A model invoice counts tokens. The job includes the attempts that failed, the tools that ran and the person who repaired the result. If those costs disappear from the comparison, the model that creates the most work for someone else can look like the best deal.
The useful measurement starts with a job that has a clear ending. "Draft a report" leaves a lot open. A report can look finished while its sources don't support its claims. Before comparing models, decide what the report has to contain and how someone will check it. Otherwise, each model gets judged against a different version of the task.
Keep the job the same
Choose a recurring task with enough real examples to show its difficult parts. Keep the examples and the available tools consistent across the comparison. Document the versions you use. A change in the input documents or the retrieval system can change the result even when the model stays the same.
NIST's research on evaluation probes for agentic AI examines whether an answer's claims are supported by its documents. For a research workflow, that is a useful distinction to keep in the evaluation. A convincing paragraph and a valid citation are different things to check.
The review should be written down before the runs start. For a report, it might require that every factual claim has a supporting source, that the reader can follow the recommendation and that the result stays within the requested scope. For another task, the checks will be different. A label on a support request has a different cost of error from a change to a customer account.
Count the work after the first answer
Record the model and tool charges for the whole attempt. Include retries that fail. Keep a record of the time someone spends checking the answer, correcting it and deciding whether it is usable. Those minutes may cost more than the difference between two model prices.
A review queue also creates delay. An answer produced in seconds can spend an afternoon waiting for the person who has to approve it. That delay matters when the workflow has a deadline, even if it doesn't appear on the model invoice.
Failed tasks need a place in the result too. Dividing the total cost by the number of accepted tasks makes failures visible in the economics. Keep the completion rate beside that figure so a small number of successful runs doesn't hide a workflow that usually needs rescue. There isn't a universal cutoff. The acceptable rate depends on what the task affects and how failure is handled.
Give the correction a name
"Human review" is a broad label. A reviewer may be checking a few facts, or reconstructing the whole report from the sources. Calling both of those review makes the second workflow look better than it is.
Name the correction when it happens. A missing source is different from a wrong conclusion. A formatting repair is different from an action that should never have been attempted. That record tells you where to improve the system. It may point to retrieval, a task definition or a permission boundary rather than to the model itself.
This is also why I wouldn't let the model grade its own work without another check. The grading method needs to be tested against examples someone has already assessed. Agreement on easy cases is useful, but it doesn't prove that the method catches the expensive mistakes.
Route only after measuring
A smaller model may be enough to classify a request. Another task may justify a stronger model and more review because a mistake costs more. Route by evidence from the task, and write down why the choice was made. Revisit it when the task or the model changes.
Start with one workflow. Follow it to an accepted result and include the work it gives back to people. Then decide what cheaper means. The price on the model page is one input to that decision; it doesn't describe the whole job.