On a recent customer project, we hit a math problem that gave us pause: if we kept feeding massive documents into top-tier, SOTA models, our operational costs were going to hit a wall - fast!
We had three choices
- transfer excessive costs to customers;
- compromise on output quality; or
- optimise the technical architecture.
We chose option three.
Our engineering team ran a sprint to overhaul how we process data.
By pairing smart model routing and payload compression with strict observability, we didn't just flatten our projected API bills, we actually made the system more accurate.
The challenge: Compounding token costs.
Early on, the easiest way to solve complex document queries was simple: throw the whole document at the most expensive model available.
It worked, but it had a massive catch. As adoption grew, our API bills scaled linearly alongside it. We needed a way to decouple top-tier performance from eye-watering inference costs.
The Fix: An intelligent AI pipeline
We ditched the basic pass-through setup and built a managed, multi-stage AI pipeline centered on four main levers:
- Workload right-sizing: Stop using a sledgehammer to crack a nut. We audited our app flows and benchmarked tasks across the entire model ecosystem. Now, every query gets routed to the cheapest model that can genuinely handle it.
- Supercharging lighter models: For mid-tier tasks, we shifted to smaller, hyper-efficient models. With some targeted prompt engineering and inference tweaks, these lighter models now reliably output premium-level results.
- Cutting the payload fat: Token waste adds up fast. Using structured frameworks (like BAML) and tighter prompt design, we compressed our input context windows and set strict boundaries on output lengths.
- Prompt caching: Why pay to process the exact same background context twice? We implemented system-level prompt caching for repeating document structures, cutting down our input token fees dramatically.
The Secret Sauce: Lower bills, zero quality loss
The trickiest part of this rebuild was avoiding the classic trap: cheaper models usually mean worse reasoning.
To ensure we didn't degrade the user experience, we replaced guesswork with a rigorous, automated evaluation loop. We curated "golden datasets" of verified queries and used an LLM-as-a-judge framework to ruthlessly grade smaller models against our high-cost baseline. A cheaper route only made it to production if it matched or beat the original output quality.
Once live, observability platforms gave us x-ray vision into our pipeline, flagging exactly where specific routing logic broke down. By reviewing these edge cases during regular development cycles and feeding them back into our benchmarks, we transformed a one-off cost-saving sprint into a continuous improvement engine.
The takeaway
You don't always need a bigger budget (or a bigger model) to get better results. Sometimes, a little architectural discipline is all it takes to make your AI leaner, faster, and smarter.
