Choose the model
after you define the work.

Write down what “good” means before comparing names.

Describe the task as a small contract: what goes in, what should come out, who relies on the result, and what a costly mistake looks like. A support draft, a code change, a source-grounded answer, and a decision aid are different jobs even when each one starts with a chat box.

Include the environment. Does the system need to read files, call tools, follow a strict output format, work in Czech, or handle a long document? A model score cannot tell you whether the entire application satisfies those requirements.

Remove options that fail a real constraint.

Documentation helps reveal what is known and what is missing. Model cards can describe intended uses, limitations, evaluation, training details, and licensing information; they are useful evidence to inspect, not a guarantee that the model fits your case.

Build a small comparison that looks like the work.

Collect examples that reflect routine requests and the cases most likely to expose a failure: incomplete context, ambiguous wording, outdated sources, unusual formatting, or a request the model should refuse or hand back to a person. Keep a set aside for a final check after you adjust prompts or configuration.

Run candidates with the same instructions, input data, tools, and output requirements. Review results without relying only on a model-graded score. For high-impact decisions, define who can review the outputs and test errors that could harm the people affected.

• Did the result solve the task, and can a reviewer verify its key claims?
• How does it behave on edge cases, conflicting evidence, and missing information?
• Does it follow the required language, format, and tool boundaries?
• How often does a person need to correct, retry, or take over?
• What happens when the model or a connected service is unavailable?

Use public benchmarks to find candidates and understand broad strengths. Stanford’s HELM work emphasizes multiple scenarios and metrics because model performance and risks cannot be summarized by one score. Then evaluate the system you intend to deploy on the work it must actually do.

Measure cost per useful task, not just price per token.

A lower request price can lose its advantage if the model needs more retries, longer prompts, extra tools, or more human correction. A self-hosted system also has hardware, energy, setup, maintenance, and monitoring costs. Record what it takes to finish the task to the agreed standard, including review time and unsuccessful attempts.

Keep quality and cost visible side by side. A cheaper model may be the right choice for a low-risk draft; a task that affects a customer, a system, or a consequential decision may require stronger checks or a different workflow.

A model choice is a maintained part of the product.

Record the chosen model and version, task, configuration, evaluation cases, limitations, and the reason it passed. Re-run the relevant checks when the model, provider, prompt, tools, source data, or workflow changes. A model that once fit the task can stop fitting as the surrounding system evolves.

For a model connected to private sources, trace the full path instead of assuming a deployment label tells the privacy story. For an agent, include permissions, recovery, and human approval in the evaluation.

For a closer look at compact models and on-device constraints, read how to evaluate a small language model ↗.

When model outputs inform decisions about people, add a task-specific review of group outcomes and error costs. See how to measure fairness in context ↗.