LLM Cost Optimization: Token Management, Caching and Model Routing

The LLM Cost Reality

LLM API pricing seems cheap per call — GPT-4o at $0.005/1K input tokens and $0.015/1K output tokens. At scale: a RAG application sending 2,000 tokens (prompt + context) and receiving 500 tokens (response) costs $0.0175 per query. At 100K daily queries: $1,750/day = $52.5K/month. Add to that: embedding API costs for RAG ($0.02/1M tokens), vector database hosting ($500-2,000/month), and serving infrastructure ($1,000-5,000/month). Total: $60-75K/month for a moderately used enterprise application. With optimization: the same application runs at $20-30K/month — the 7 levers described below reduce cost 50-70% without reducing quality.

Lever 1: Token Optimization — Shorter Prompts, Same Quality

Every token costs money. Token optimization techniques: system prompt compression (the 500-token system prompt that says "You are a helpful assistant that answers questions about our company's products and services using the provided context. Always be professional and accurate..." can be compressed to 150 tokens: "Answer product questions using context. Be accurate and professional." — 70% fewer tokens, same behavior), context window management (for multi-turn conversations: summarize earlier turns instead of sending the full conversation history. Turn 1-5 summarized in 200 tokens instead of sent verbatim at 2,000 tokens), retrieval optimization (send the top 3 most relevant chunks instead of top 10 — fewer context tokens with minimal quality impact. The top 3 chunks contain the answer 85% of the time), and output length control (set max_tokens to the expected response length — a factual answer needs 100 tokens, not 500). Combined impact: 25-40% token reduction across the application.

Lever 2: Semantic Caching — Don't Call the LLM Twice for the Same Question

Semantic caching: when a user asks a question similar to a previously answered question, return the cached response instead of calling the LLM. Implementation: embed the user query → search the cache for similar queries (cosine similarity > 0.95) → if match found, return cached response → if no match, call LLM and cache the response. Cache hit rates for enterprise applications: IT helpdesk (60-70% — many users ask the same questions), product FAQ (50-60% — common questions about features and pricing), HR policy bot (40-50% — seasonal patterns in questions), and customer service (30-40% — more diverse queries). Implementation: Redis with vector search capability, or a dedicated semantic cache (GPTCache, LangChain cache). Cache invalidation: when the knowledge base updates, invalidate cached responses that referenced updated documents. Cost impact: 30-60% cost reduction depending on query diversity.

Lever 3: Model Routing — Right Model for Each Query

Not every query needs GPT-4o ($0.01/query). Many queries work equally well with GPT-4o-mini ($0.001/query). Model routing: classify each query by complexity → route to the appropriate model. Classification approaches: keyword-based (queries with specific keywords indicating complexity: "compare," "analyze," "explain the implications" → GPT-4o; simple queries: "what is," "how do I," "where can I find" → GPT-4o-mini), intent-based (use a lightweight classifier to categorize query intent → map intents to model tiers), and fallback routing (try GPT-4o-mini first → if confidence is low or user feedback is negative → retry with GPT-4o). Typical distribution: 60-70% of queries route to the cheaper model, 30-40% to the premium model. Cost impact: 50-65% reduction in LLM API costs.

Lever 4: Batch Processing — Bulk Calls at Discount

OpenAI and Azure OpenAI offer batch processing at 50% discount for non-real-time workloads. Applications that can use batch: document processing (summarize 1,000 documents overnight), data enrichment (classify 50,000 records), content generation (generate product descriptions for 5,000 SKUs), and evaluation (evaluate model quality on 10,000 test cases). Batch processing is asynchronous — submit the batch, receive results hours later. Not suitable for: real-time chat, interactive applications, or time-sensitive responses. But for the 20-30% of LLM workloads that are batch-eligible: 50% cost reduction with zero quality impact.

Lever 5: Fine-Tuning — Smaller Model, Same Accuracy

Fine-tuning a smaller model (GPT-4o-mini, Llama 3, Mistral) on your domain data can achieve 90-95% of the quality of the larger model at 5-10x lower cost. When fine-tuning works: the task is specific and repetitive (classifying support tickets, extracting invoice fields, generating structured reports), you have 500-5,000 labeled examples, and consistency of output format matters more than creative generation. When fine-tuning doesn't work: the task requires broad knowledge (general Q&A), few examples exist (under 100), or the task changes frequently (the fine-tuned model becomes outdated). Fine-tuning cost: $5-50 to train (one-time), then inference at the smaller model's rate — typically 5-10x cheaper than the large model. Combined with model routing: fine-tuned small model handles 70% of queries, large model handles 30%.

Lever 6: Local/Open-Source Models — Eliminate API Costs

Open-source models (Llama 3, Mistral, Phi-3) run on your own infrastructure — eliminating per-token API costs entirely. The trade-off: infrastructure cost (GPU hosting: $1-8/hour per GPU) vs API cost. Breakeven analysis: at 500K+ daily queries, self-hosted open-source models cost less than API calls. Below 100K daily queries, API is cheaper (no infrastructure overhead). Self-hosted advantages: no data leaves your infrastructure (compliance requirement for some industries), no rate limiting, and predictable costs (fixed monthly infrastructure, not variable per-token). Self-hosted challenges: model management (updates, monitoring, scaling), infrastructure operations (GPU management, Kubernetes, availability), and quality gap (open-source models are 80-90% as capable as GPT-4o for most tasks — acceptable for some applications, not for others).

Lever 7: Cost Monitoring and Budgeting

LLM cost monitoring: track per-request: model used, input tokens, output tokens, latency, and cost. Aggregate: daily cost by application, cost per user, cost per conversation, and cost trend. Alerts: daily cost exceeds budget → alert. Per-user cost exceeds threshold → investigate (one user generating 10,000 queries/day?). Cost spike → investigate (did a prompt change increase token count?). Budget controls: per-application monthly budget with hard limit (application returns cached/degraded response when budget exhausted rather than exceeding), per-user rate limiting (prevent abuse), and model tier limits (restrict GPT-4o usage to applications that justify the cost). Tools: LangSmith (LangChain observability), Helicone (LLM proxy with cost tracking), Azure OpenAI usage dashboard, and custom monitoring via API logging.

Cost Optimization Case Study: Enterprise Knowledge Assistant

An enterprise knowledge assistant serving 2,000 employees: before optimization — all queries to GPT-4o, no caching, full system prompt (500 tokens) + 5 RAG chunks (2,000 tokens) per query, average 500 token response. Cost per query: $0.0175. Daily volume: 5,000 queries. Daily cost: $87.50. Monthly cost: $2,625. After optimization (7 levers applied) — Lever 1 (token optimization): compressed system prompt to 150 tokens, reduced to 3 RAG chunks (1,200 tokens). Cost per query: $0.012 (31% reduction). Lever 2 (semantic caching): 45% cache hit rate. Effective queries to LLM: 2,750/day. Lever 3 (model routing): 60% of queries routed to GPT-4o-mini ($0.001). Lever 4 (batch processing): 500 daily document processing queries batched at 50% discount. Combined effect: daily cost from $87.50 to $22.50. Monthly cost from $2,625 to $675. 74% cost reduction with no measurable quality decrease. The optimization took 2 weeks to implement. Payback: immediate.

LLM Cost Comparison: Provider and Model Selection

Provider selection strategy: For most enterprise applications: GPT-4o-mini or Claude 3.5 Haiku as the default model (80-90% of queries), GPT-4o or Claude 3.5 Sonnet for complex queries (10-20%). Multi-provider strategy: use model routing to select the best model per query — not just the cheapest, but the one that produces the best quality for that query type at the lowest cost. This avoids: single-provider dependency (outage affects all AI applications) and overpaying (using premium model for simple queries).

Building the Cost Optimization Pipeline

Implement cost optimization in layers — each layer adds savings: Layer 1 (Week 1-2): Monitoring (instrument every LLM call with: model, tokens, cost, latency. Build the cost dashboard. This alone reveals: which queries are expensive, which models are overused, and where the easy wins are). Layer 2 (Week 3-4): Token Optimization (compress system prompts, reduce RAG context chunks, set appropriate max_tokens. Savings: 25-35%). Layer 3 (Week 5-6): Caching (deploy semantic cache with Redis + vector search. Start with aggressive caching (similarity threshold 0.92), tune based on quality feedback. Savings: 25-45% additional). Layer 4 (Week 7-8): Model Routing (build query classifier, route simple queries to mini model. Start with keyword-based routing, graduate to ML-based classification. Savings: 40-60% additional). Total implementation: 8 weeks. Total cost reduction: 60-75%. Each layer is independent — implement in any order based on which provides the fastest payback for your traffic pattern. The monitoring layer is always first — you can't optimize what you can't measure.

Prompt Caching: Provider-Level Optimization

Anthropic and OpenAI now offer prompt caching at the provider level: if the system prompt (or any prefix of the input) was sent recently, the provider caches it and charges reduced rates on subsequent calls. Anthropic prompt caching: 90% discount on cached input tokens. OpenAI: 50% discount. This is different from semantic caching (which caches the complete response): prompt caching reduces the cost of the system prompt and RAG context that's repeated across queries. For an application with a 1,000-token system prompt sent with every query: prompt caching saves $0.004 per query at GPT-4o rates. At 100K daily queries: $400/day = $12K/month saved — from a provider feature that requires zero code changes (just consistent prompt prefixes). Implementation: structure prompts so the system message and static context appear first (cached), and the user query appears last (not cached). This ordering maximizes cache hit rate. Prompt caching + semantic caching + model routing together: 70-80% total cost reduction — making enterprise LLM applications financially viable at any scale.

Building a Cost-Optimized LLM Architecture: Reference Design

Combined savings example: Baseline: all queries to GPT-4o = $10,000/month. After routing (80% to mini): $3,400/month. After caching (30% hit): $2,380/month. After prompt optimization (20% reduction): $1,904/month. Total: $1,904 vs $10,000 baseline = 81% reduction. Implementation: 2-3 weeks. The $8,000/month savings pays for engineering in the first week.

LLM Cost Forecasting: Planning for Growth

LLM costs scale with usage: user growth (3x users = 2-3x tokens after caching). Feature expansion (each new LLM feature adds consumption — forecast per-feature cost before building). Model evolution (newer models are cheaper — GPT-4o-mini is 33x cheaper than GPT-4. Budget current pricing but expect reductions). Budget model: Monthly LLM cost = daily_active_users times avg_interactions times avg_tokens times price_per_token times (1 minus cache_hit_rate) times routing_multiplier. Track monthly. If actual exceeds forecast by 20%+: investigate for unexpected patterns or missing optimizations.

Go Deeper

Continue building your understanding with these related resources from our consulting practice.