Your AI bill is a routing problem

An AI feature that costs a few cents per request is a rounding error in a pilot and a line item in production. The usual response is to negotiate rates or switch providers. The larger savings are usually in the architecture.

The problem: one model for everything

Look at the traffic of a typical AI product and a pattern appears. A large share of requests are routine: classify this, extract that, answer a question the documents answer directly. A minority need real reasoning. Yet most systems send everything to the same frontier model, often with the same long system prompt rebuilt on every call.

Four levers, in order of effort

1. Cache the shared prefix. Long system prompts, tool definitions and reference documents are often identical across requests. Providers including Anthropic and OpenAI discount cached input tokens heavily and serve them faster. Structuring prompts so the stable part comes first is often an afternoon's work.

2. Batch what can wait. Nightly enrichment, evaluations, back-fills and report generation do not need an answer in a second. Batch APIs run them at a substantial discount.

3. Route by difficulty. Send each request to the cheapest model that handles that kind of request well enough, and escalate when it does not. Research such as FrugalGPT and RouteLLM showed that cascades and learned routers can match a strong model's quality at a fraction of the cost on suitable workloads. A router can be as simple as rules by request type, or a small classifier trained on which model succeeded before.

4. Distil the hot path. For the single highest-volume task, train a small model on the frontier model's outputs. Work on distillation, such as Google's "distilling step-by-step", shows small students can match much larger models on narrow tasks. The upfront cost is real: you need labelled data and an evaluation set. The running cost drops sharply and latency improves.

The catch: you cannot route what you cannot measure

Every lever trades some risk of lower quality for cost. Without an evaluation set per request type, you will either save money and quietly break things, or never dare to change anything. So the order of work is:

1. 01Trace production traffic and group requests by type.
2. 02Build a small evaluation set for each high-volume type.
3. 03Score candidate models on each, and record cost and latency.
4. 04Route each type to the cheapest model that passes, with an escalation path.
5. 05Re-run the evaluation on every model or prompt change.

A note on self-hosting

Running open models on your own GPUs can be cheaper at steady, high volume, and sometimes is required for data reasons. At low or spiky volume it is usually more expensive than an API. Model it with your real traffic before buying hardware.

What to measure

• Cost per task type. Not per token: per resolved request, by type, so a cheaper model that needs retries shows up honestly.
• Quality per route. Evaluation scores for each request type on each model it can be routed to.
• Escalation rate. How often the cheap route hands off to the expensive one; rising escalation eats the saving.
• Cache hit rate and latency. The share of input tokens served from cache, and the response time users see.

How we would start

Trace a week of production requests, tag them by type and price each type. The biggest saving is usually obvious from that table. This is part of our LLMOps work, and it leads naturally to LLM fine-tuning and distillation or private deployment where the numbers justify it. ClearLoad, which routes work between people and models by complexity and cost, applies the same idea one level up.