Here's a pattern we see constantly: a company builds an AI feature, picks GPT-4o or Claude Sonnet because "we want the best quality," ships it to production, and then watches their monthly bill climb to €5,000, €10,000, €20,000. Nobody knows exactly where the money is going. Nobody has measured whether the expensive model is actually necessary.
Sound familiar? You're not alone. Most organizations we work with are overspending on AI by 50-80%. Not because AI is inherently expensive—but because they haven't optimized.
The Hidden Cost Problem
LLM costs are invisible by default. Unlike cloud compute where you can see CPU and memory usage, token consumption is buried in API responses that most teams never log. This creates a blind spot:
- Which feature is eating most of the budget?
- Which users are driving the highest costs?
- Are identical queries being processed repeatedly?
- Could a cheaper model handle 80% of the workload?
Without answers to these questions, you're flying blind.
The Four Most Expensive Mistakes
After auditing dozens of AI implementations, we've identified the patterns that drain budgets:
1. Using flagship models for everything
GPT-4o costs $5 per million input tokens. GPT-4o-mini costs $0.15—that's 33x cheaper. For many tasks (classification, extraction, simple Q&A), the smaller model performs identically. Yet most teams default to the expensive option "just to be safe."
2. No prompt caching
Both OpenAI and Anthropic offer prompt caching that can reduce costs by up to 90% for repeated context. A 10,000-token system prompt sent with every request? That's pure waste if the same instructions are being processed over and over.
3. Bloated prompts
We've seen prompts with 4,000 tokens of instructions when 800 would suffice. Every unnecessary token multiplied by thousands of daily requests adds up fast.
4. Duplicate processing
The same document analyzed multiple times. The same question answered repeatedly. Without response caching, you're paying full price every single time.
The Solution: Intelligent Model Routing
The fix isn't to stop using AI—it's to use the right model for each task. This is called model routing, and it works like this:
- Simple tasks (classification, extraction, formatting): GPT-4o-mini, Claude Haiku, or self-hosted Llama/Mistral
- Medium complexity (summarization, translation): GPT-4o-mini or Claude Sonnet
- Complex reasoning (analysis, planning, code generation): GPT-4o, Claude Sonnet, or Claude Opus
Tools like LiteLLM make this easy to implement. You define routing rules, and requests automatically go to the appropriate model based on task type, complexity, or custom logic.
The Open-Source Alternative
Here's where it gets interesting for cost-conscious teams: open-source models like Llama 3 and Mistral can handle many tasks at near-zero marginal cost when self-hosted.
Yes, there's infrastructure cost. But for high-volume workloads, a €500/month GPU server running Llama can replace €5,000/month in API calls. We've seen Dutch companies cut their AI costs by 70-90% by moving classification and extraction tasks to self-hosted models.
The bonus: your data never leaves your infrastructure. For GDPR-conscious organizations, this solves two problems at once.
Start with Visibility: Langfuse
Before you can optimize, you need to measure. Langfuse is an open-source LLM observability platform that tracks every call, token count, and cost. Within a few days of implementation, you'll have answers to all those blind-spot questions:
- Cost breakdown per feature, user, and model
- Token usage patterns over time
- Duplicate query detection
- Prompt size analysis
This visibility alone often reveals quick wins worth thousands per month.
Real Results
Here's what optimization looks like in practice:
A mid-sized insurer was spending €8,000/month on flagship models for claims intake. Our audit revealed that 85% of queries were simple classification tasks. By routing these to GPT-4o-mini and a self-hosted Llama model, costs dropped to €2,400/month—a 70% reduction with identical output quality.
A law firm had €4,000/month in LLM costs with zero visibility. Langfuse tracing uncovered that the same contracts were being analyzed repeatedly (no caching) and system prompts were being re-sent with every request. After implementing caching and prompt optimization, costs dropped to under €1,000/month.
Getting Started: Six Steps to Lower AI Costs
If you're running LLMs in production and haven't optimized, you're almost certainly overspending. The good news: the fixes are straightforward, and the ROI is immediate. Here's the playbook:
Step 1: Get visibility with observability
You can't optimize what you don't measure. Implement Langfuse or similar tracing to log every API call, token count, and cost. Within a week, you'll have answers to the crucial questions: which features cost the most? Which users? Are queries being repeated? This data forms the foundation for everything that follows.
Step 2: Identify your quick wins
Analyze your data for the biggest cost drivers. Look for patterns: repeated identical queries (caching opportunity), simple tasks on expensive models (routing opportunity), or bloated prompts (optimization opportunity). Sort by impact—tackle what yields the most savings first.
Step 3: Implement model routing
This is where the big savings live. Classify your use cases by complexity and route them to the appropriate model. Use tools like LiteLLM to define routing rules. Start conservative: test the cheaper model on a subset of your traffic and compare output quality before fully switching over.
Step 4: Enable caching
Implement prompt caching for repeated context (system prompts, examples) and response caching for identical queries. OpenAI and Anthropic offer native prompt caching; for response caching, you can use Redis or similar solutions. This alone can reduce costs by 30-50%.
Step 5: Use batch processing for non-urgent tasks
Not everything needs to be real-time. Document analysis, bulk classification, content generation for later—these tasks can be processed in batches. OpenAI's Batch API offers a 50% discount for requests completed within 24 hours. Identify which workloads can wait and shift them to off-peak or batch processing. This not only saves money but also reduces peak load on your systems.
Step 6: Optimize your prompts
Review your prompts critically. Remove redundant instructions, consolidate examples, and test whether shorter versions deliver the same quality. Reducing a prompt from 4,000 to 1,000 tokens saves 75% on that component—multiplied by thousands of daily requests, this adds up fast.
Or let us do it for you. We offer free cost audits that show exactly where you're overspending and how much you can save. No commitment, results in one week.