Generative AI Development, Trade-Offs Written Down

01

Prompting, retrieval, or fine-tuning

Prompting is free to change and should always be tried first, because a well-built system message with a few worked examples solves more than people expect. Retrieval is the answer when the model needs facts it was never trained on, or facts that change every week. Fine-tuning buys a consistent voice, a rigid output format and cheaper tokens on a narrow task, and it is the only one of the three that goes stale the day your content changes.

What it does to the bill: Prompting is cheapest to build and most expensive per token at scale, retrieval adds an index to maintain, fine-tuning adds a training run every time the domain moves.

02

Hosted API, or open weights on your own hardware

A hosted API reaches production faster and gives you frontier quality with no infrastructure to run. Open weights on your own servers means no text leaves your network and a fixed monthly cost instead of a per-token one, plus a GPU bill whether you use it or not. The crossover is volume and compliance, not ideology.

What it does to the bill: We model the break-even against your expected traffic before recommending either, because it moves every time a provider changes its pricing.

03

An agent, or a pipeline that happens to call a model

An agent decides its own next step. That earns its keep when the path changes per request, and becomes a liability when it does not, because every extra step multiplies latency, cost and the number of ways it can go wrong. Most processes we are asked to turn into agents are a fixed sequence with two or three model calls in it, and that version is cheaper, testable and far easier to debug. Where an agent really is right, it ships with a step ceiling, idempotent tools, and a human approving anything that spends money or sends a message.

What it does to the bill: An agent loop costs you every token it spends thinking, including the runs that end up going nowhere.

04

One large model everywhere, or routing by difficulty

Sending every request to the biggest available model is the default, and it is usually the wrong default. Classification, extraction and routing steps often run on a much smaller and cheaper model at the same measured accuracy, which is exactly what the evaluation set is for: it lets you downgrade a step and prove nothing broke. Caching repeated queries and trimming context get less attention than model choice, and often save more.

What it does to the bill: This is the fork with the largest gap between the lazy answer and the right one.

01

The answer has to be identical every time

Pricing, tax, payroll, eligibility, compliance. A language model produces plausible output, not repeatable output. If a wrong answer creates a liability, the logic belongs in code with tests around it, and the model at most explains the result in plain English.

02

There is nothing to ground it in

Retrieval only works if the knowledge exists somewhere. If your policies live in three people’s heads and a WhatsApp group, the first project is writing them down. We are happy to build the retrieval layer afterwards, but no model can retrieve what was never written.

03

A query, a rule or a report already answers it

A surprising share of AI briefs are really a search box, a saved filter, or a scheduled email. Those cost less to build, cost nothing per request, and never invent anything. When that is the honest answer, that is the answer you get.

04

The output is a number, a score or a label

Forecasting demand, scoring churn risk, spotting a defect in a photo, sorting tickets into categories. That is classical machine learning, it needs labelled data, and it belongs on our machine learning page rather than this one.