How to choose an LLM for your project

Choosing a large language model is a business decision with a technical test attached. This guide gives you a repeatable way to make it: define the job, weigh ten criteria, score your shortlist in a matrix and test it on your own data. It closes with starting points for five common use cases as of October 2026 and a checklist for procurement.

Start with the job, not the model

Teams that start from a leaderboard tend to buy more model than they need, or the wrong kind. Start by writing a one-paragraph task brief that anyone on the project would agree with.

• Input: what goes in, such as support tickets, contracts, code diffs or call transcripts, and how long a typical and a worst-case input is.
• Output: what a good result looks like, ideally with three real examples.
• Volume: requests per day now and in a year, and the peak hour.
• Stakes: what a wrong answer costs, and who catches it.
• Constraints: data that can't leave your network, required regions, budget ceilings and latency targets.

The brief usually rules out half the market before you compare a single benchmark. A task that handles patient records on-premises and a task that drafts marketing copy shouldn't share a shortlist.

The ten decision criteria

These criteria cover nearly every model decision. For each one, the table gives the question to ask and how to get a real answer instead of a vendor claim.

A few details trip people up. Latency has two parts: time to the first token, which users feel in chat, and total generation time, which dominates batch jobs. Reasoning models can add many seconds of thinking before the first visible token, and some providers sell speed at a premium; OpenAI, for example, offers a Fast mode for GPT-6 Astra at twice the standard price.

Context length has a pricing wrinkle. As of October 2026, OpenAI's GPT-6 models and Google's Gemini 3.1 Pro charge more per token above a prompt-length threshold, and Claude Haiku 5.5's rates rise fivefold for prompts over 100,000 tokens. Anthropic's other current models charge the same rate across their full one-million-token window. Measure your real input sizes with the token counter before you assume a window or a price.

A weighted scoring matrix

A scoring matrix turns a debate into arithmetic. Agree on weights before you see results, score each candidate from 1 to 5 per criterion, and multiply.

Here's an example for an internal assistant that answers staff questions about company policies, some of which contain personal data. The candidates are a top-tier closed API model, a mid-tier closed API model and a mid-sized open-weight model hosted in your own cloud account.

Model A wins on quality but loses on cost and lock-in, and the self-hosted model edges out the mid-tier API. Now run a sensitivity check. If the team decides privacy matters less, say because the policy documents are scrubbed of personal data, moving 10 points from privacy to quality changes the totals to 3.70 for A, 3.80 for B and 3.55 for C, and B becomes the winner.

Run a quick evaluation on your own data

Public benchmarks tell you which models are generally strong, not which one handles your inputs. A focused evaluation takes a few days and is the best predictor of production performance you can get.

Build a golden set

Collect 50 to 200 real, anonymized inputs. Include the common cases in proportion, plus the edge cases and known failures that worry you, and write a reference answer or explicit pass criteria for each.

Keep the set frozen and versioned. It becomes your regression test for every prompt change, model update and provider switch.

Choose a scoring method

Match the scoring to the task. Extraction can be scored field by field against the reference, code can be scored by running unit tests, and classification by simple accuracy.

Open-ended outputs, such as answers and drafts, need a rubric: for instance, 1 to 5 for correctness, completeness, grounding in the provided sources and tone. Have two people score the same 20 outputs first, to make sure the rubric produces consistent judgments.

Use LLM-as-a-judge, carefully

Scoring hundreds of outputs by hand is slow, so many teams use a strong model as the grader, an approach called LLM-as-a-judge. A 2023 study by Zheng and colleagues found that strong judges agreed with human preferences more than 80% of the time, roughly matching agreement between people.

The same study documented biases toward the first answer shown, toward longer answers, and toward the judge's own outputs. Swap the order of answers in pairwise comparisons, give the judge your rubric, and prefer a judge from a different model family than the candidates. Then check its scores against 30 to 50 human-labeled examples before you trust it.

Measure cost per successful task

Log the actual input, output and reasoning tokens from each API response rather than estimating them. Then fold in what failures cost you, because a cheap model that fails often isn't cheap.

Here's a worked example. Each task uses 3,000 input and 500 output tokens. An upper-mid-tier model priced at $4 and $20 per million tokens costs $0.012 plus $0.010, or $0.022 per task, and fails 3% of your golden set. A budget model at $0.10 and $0.50 costs $0.0003 plus $0.00025, or $0.00055 per task, but fails 9% of the time.

If each failure costs $2 of staff time to catch and fix, expected cost per task is $0.022 + (0.03 × $2) = $0.082 for the first model and $0.00055 + (0.09 × $2) = $0.18 for the second. The model that's 40 times cheaper per call is more than twice as expensive per task. The LLM cost calculator handles the token side of this math at your volumes.

Starting points for five common use cases

The table below describes what to shortlist by tier and traits, with example candidates as of October 2026. Model names change every few months, but the traits that matter for each job are much more stable.

For support, most traffic is routine, so a fast model with good retrieval handles the bulk and a stronger model takes the escalations. For coding, the top labs trade the lead on agentic coding benchmarks from one release to the next, so trial two candidates on your own repository. For document Q&A, a long context window doesn't replace retrieval-augmented generation on large collections, because sending a million tokens with every question is slow and expensive.

Extraction is where small models shine. Combine a schema, a low temperature and batch processing, which Anthropic and Google price at half their standard rates, and route low-confidence records to a larger model or a person. For drafting, quality is partly taste, so a blind side-by-side with the people who'll edit the output beats any benchmark.

Deployment routes and vendor risk

The same model can often be bought three ways. The route you choose affects price, data handling and how quickly you get new features.

• Direct from the provider: fastest access to new models and features, with the provider's own data terms. OpenAI, for example, offers zero data retention to eligible API customers.
• Through a cloud marketplace: you use your existing cloud contract, identity controls and billing. Anthropic lists its Claude models on Amazon Bedrock, Google Cloud Vertex AI and Microsoft Foundry, and OpenAI lists GPT-6 Astra on Microsoft Azure and AWS Bedrock. Residency has a price: Anthropic notes that regional endpoints on Bedrock and Vertex AI cost 10% more than global ones.
• Self-hosted open weights: maximum control over data and versions, but you own scaling, patching and evaluation. Our guide to open-source vs closed LLMs covers this trade-off in depth.

Plan for churn. Google's Gemini 3.7 Flash became generally available on August 13, 2026, and was deprecated on October 8, 2026, with requests routed to Gemini 3.8 Flash. Automatic routing like that keeps applications running, but behavior can shift underneath you.

Protect yourself with a thin internal interface between your application and any provider SDK, pinned model versions where providers allow it, prompts and golden sets stored in version control, and a tested fallback model. Then a switch costs a re-run of your evaluation rather than a rewrite.

Procurement checklist

Before you sign, get written answers to these questions. They apply to direct APIs and cloud marketplaces alike, and this is general guidance rather than legal advice.

Common mistakes

These errors show up again and again in model selection. Each is cheap to avoid and expensive to discover in production.

• Choosing from a leaderboard. Rankings average many tasks that aren't yours.
• Comparing price per token. Cost per successful task, including failures and reasoning tokens, is the number that matters.
• Testing on clean demo inputs. Real inputs are longer, messier and stranger, so build the golden set from production data.
• Ignoring tail latency and rate limits. Averages hide the slow responses users remember.
• Hard-wiring one provider. SDK-specific code and prompt quirks scattered through the codebase turn a model switch into a project.
• Never re-testing. Re-run your evaluation whenever a provider updates or reroutes a model.

Next steps

For a quick shortlist, the LLM picker recommends models from a few questions about your use case and budget, and side-by-side comparison lines up prices, context windows and features. If self-hosting is on the table, read open-source vs closed LLMs next. And if you'd like someone to run the evaluation with your team, you can get expert help.