LLM Cost Control in Production: Routing, Caching, and Limits

Finance forwards a usage screenshot. The feature has been live for three weeks. Nobody owns a budget, and every request still hits the frontier model because that is what the prototype used. Familiar?

Demo traffic is cheap. Production traffic is not. Teams that ship without cost controls discover the bill after users love the feature.

Measure before you optimise

Track per request:

• Model and provider
• Input / output tokens
• Cache hits
• Tool-call fan-out (agents multiply cost)
• Feature or tenant ID

If you cannot attribute spend, you cannot control it. Dashboards without feature and tenant dimensions are decoration.

Levers that work

1. Model routing

Reserve frontier models for hard cases. Use smaller or cheaper models for classification, extraction, and easy Q&A. Route by task type and confidence, not by habit. Habit is how every call becomes expensive.

2. Caching

Cache embeddings, retrieval results, and identical prompt+context responses where freshness allows. Semantic cache helps. Start with exact cache for hot paths. Boring caches pay the rent.

3. Context discipline

Long contexts are silent cost multipliers. Retrieve less, rerank, and drop low-value chunks. Summarise histories instead of appending forever. Every token you send is a token you pay for, twice if you also generate a long answer.

4. Hard limits

Per-tenant quotas, max tool iterations, max tokens per request, and circuit breakers when budgets trip. Soft warnings are not enough. Soft warnings get muted.

5. Batch where latency allows

Offline enrichment and report generation should not use interactive pricing paths. Separate the work that must be fast from the work that must be cheap.

Agents need extra care

Each tool call can trigger more model usage. Cap steps, prefer deterministic tools, and log cost per trajectory. See observability for agents. An agent that “thinks” for twelve steps on every ticket will teach you economics the hard way.

Quality vs cost

Never cut cost by deleting evals. Cut cost by routing and caching, then re-score the golden set. If quality drops below the bar, the saving is fake. You traded money for quiet failures.

Ops ownership

Cost is an ops metric alongside latency and quality. Managed AI operations includes this loop after launch. Someone has to watch the curve, not only ship the feature.

Checklist

Keep this next to your production readiness list:

• Per-feature cost dashboards
• Routing policy documented
• Cache on hot paths
• Quotas and circuit breakers
• Agent step caps
• Eval gate after cost changes

If spend is climbing and ownership is unclear, discuss your use case.