The problem

Our client, a B2B SaaS company in the legal sector, crossed $40,000/month on OpenAI costs in February 2026. They were running GPT-4o for every single user request — document summarisation, clause extraction, Q&A, drafting, the works. Management wanted to cut the bill in half. We got it down by 60% in two weeks without a single accuracy regression.

Here's exactly what we did, in order of ROI.

Step 1 — Prompt caching (immediate 30% reduction)

The first thing we audited was prompt structure. Almost every call was structured as:

[Long system prompt — 800 tokens]
[Few-shot examples — 1,200 tokens]
[User message — variable, avg 400 tokens]

The system prompt and examples were identical across every call. OpenAI's prompt caching (available on GPT-4o and GPT-4o-mini as of late 2024) automatically caches prompt prefixes that exceed 1,024 tokens. Cached tokens cost 50% less on input.

Change: We restructured all prompts to put the static system prompt + examples first, followed by the dynamic user content. That's it. No code change beyond prompt ordering.

Result: 30% cost reduction on the first day. The cache hit rate settled at around 78%.

Step 2 — Model routing by task complexity

Not every task needs GPT-4o. We categorised the client's 12 distinct call types by complexity, accuracy tolerance, and output length.

After a week of shadow-testing, we routed:

Task Before After
Document summarisation GPT-4o GPT-4o-mini
Clause extraction (structured) GPT-4o GPT-4o-mini
Legal Q&A (complex) GPT-4o GPT-4o (kept)
Contract drafting GPT-4o GPT-4o (kept)
Internal search reranking GPT-4o text-embedding-3-small

GPT-4o-mini costs roughly 15x less than GPT-4o. We routed ~65% of volume to it.

Accuracy check: We ran 500 human-evaluated samples per task type. Summarisation and clause extraction showed no meaningful quality difference. Q&A and drafting stayed on GPT-4o.

Result: Additional 20% reduction.

Step 3 — Request batching for async workflows

The client had a nightly job that ran 2,000 individual API calls to generate weekly summaries. Each was a separate HTTP request.

We rewrote the job to use OpenAI's Batch API, which processes requests within 24 hours at 50% of the standard price. The nightly summary job is async by nature — nobody reads it until morning — so the latency trade-off was irrelevant.

Result: 50% reduction on that specific workload, which was ~12% of total spend.

Step 4 — Output token trimming

We audited the actual outputs being consumed downstream. In 8 of the 12 task types, the application was only using the first paragraph or a structured JSON block.

We added explicit constraints to every prompt:

  • "Respond in JSON only. No prose before or after."
  • "Maximum 150 words."
  • Lowered max_tokens from the default to realistic values per task type

Average output token count dropped from 340 to 180 tokens per call.

Result: ~10% additional reduction across the board.

Final numbers

Optimisation Reduction
Prompt caching ~30%
Model routing ~20%
Batch API for async jobs ~12% of total
Output token trimming ~10%
Total ~60%

Starting bill: $40,200/month
After sprint: $16,100/month
Saving: $24,100/month ($289,200/year)

The principle

The single biggest mistake teams make with LLM cost is treating the model as a monolith. Every call is different. Route them accordingly, structure your prompts for caching, and set explicit output constraints. You will almost always find 40-60% savings without touching a single accuracy metric.