AR Academy Library · Lesson 5 of 5 · $4,999 for all 5, waived on scoping call
EnrollRequest routing with caching and model tiering
Where AI costs actually accumulate
The bill is not where teams expect. It is not the model calls in your marquee feature — it is the retries, the retrieval context you pass on every request, the summarization step nobody profiled, and the background evaluation runs. Instrument before you optimize. Log input tokens, output tokens, model, endpoint, and request purpose. The first cost review will surface two or three code paths responsible for 70% of spend. Fix those first. Everything else is noise.
Prompt caching as the biggest lever
Anthropic and OpenAI both ship prompt caching — you pay a cache write once, then reads are ~10% of the cost. For any workload with a stable system prompt, retrieved context, or few-shot examples, caching cuts input token cost dramatically. Structure prompts with stable content first (system, retrieved docs, examples), then the variable user input at the end. Cache TTLs are short (5 min on Anthropic) — cache what actually gets reused, not what could theoretically be reused.
Model routing — Haiku, Sonnet, Opus tiers
Route requests by complexity, not by default. Simple classification, extraction, or filtering: Haiku (cheap, fast, good enough). Standard reasoning, agentic loops, code generation: Sonnet. Deep multi-step reasoning, capstone drafts, hard tool use: Opus. A router that classifies requests before dispatch — even a Haiku-based classifier — pays for itself in production. Measure quality per tier. If Haiku's output rate of user-fixed edits is under 10%, you're overpaying by using Sonnet.
Streaming, batching, and async patterns
Streaming reduces perceived latency but not cost. Batching (Anthropic and OpenAI batch APIs) offers ~50% discount for non-real-time workloads — evals, offline enrichment, bulk generation. Async patterns matter for parallel tool calls: run independent tool invocations concurrently, then aggregate. And for long-running agents, checkpoint state so you can resume from a partial trajectory instead of paying for the full context on retry.
Cost monitoring and budgets
Every production AI system needs per-tenant budgets, per-request cost tracking, and daily cost alerts. Cost anomalies indicate real bugs — an infinite loop in an agent, a retrieval bug pulling 100x context, a prompt injection extracting your entire knowledge base. Set hard budgets by feature and by tenant. Fail loud when they hit; don't silently degrade. The teams that survive high AI bills are the ones that treated cost as a first-class signal, not a monthly surprise.
Enroll — 30 min scoping call.
We build what these lessons describe. $4,999 library, waived when we scope work.
