Agent Cost & TCO: Token Economics and Latency Budgets

Per-token prices have fallen for years, yet agent bills keep rising. The reason is structural: an agent re-reads its own growing context on every step of its loop, so a single task can consume far more tokens than the one model call you imagine. The price per token is the number everyone watches and the one that matters least. This deep dive is about the numbers that actually set the bill — and the majority of cost that never appears on the model invoice at all.

1Why cheaper tokens produce bigger bills

Start with the paradox that confuses every budget conversation: the price of a token has fallen dramatically over the past few years, and yet enterprise AI bills have climbed over the same period. If cost tracked the rate card, the bill should be shrinking. It isn't. The objective of this section is to explain why — because the explanation is the whole foundation of agent economics.

The resolution is that an agent's cost is not rate × (one call). It is rate × tokens-per-step × steps × runs. The rate is one factor of four, and it is the only one falling. The other three — how big each step's context is, how many steps a task takes, and how many times the agent runs — all grow as you make agents more capable. A single agentic task can consume far more tokens than the one model call people picture when they think "an LLM answers a question." Cheaper tokens have actually encouraged this: lower rates make teams comfortable with longer prompts, more tool calls, more retries, and deeper loops — so total usage grows faster than the price drops.

The rate card is the one number falling and the one number everyone watches. The bill is set by the three numbers nobody puts on a slide.

2The dominant driver: the loop re-reads itself

The previous section said the multiplier dominates; this one names the single biggest part of it. If you fix one thing about agent cost, fix this — because it is the mechanism that turns a cheap model call into an expensive task, and it is invisible until you trace it.

Recall the shape of the agent loop: reason, call a tool, observe the result, repeat. The cost trap lives in the word repeat. On every iteration, the agent re-sends its entire growing context to the model — the system prompt, the tool definitions, the original goal, and the full transcript of everything it has done so far. Step 1 sends a small context. Step 2 sends step 1's context plus step 1's output. Step 10 re-sends everything from steps 1 through 9. The conversation does not cost N for N steps; it costs roughly N-squared, because each new step re-pays for all the context before it.

Two consequences follow, and both shape where you optimize. First, input tokens dominate the bill — the agent reads vastly more than it writes, because it re-reads its history constantly. Second, anything that bloats per-step context is multiplied by every future step: a verbose system prompt, oversized tool definitions, an un-compacted transcript. Trimming 1,000 tokens from a context that gets re-sent ten times saves 10,000 tokens, not 1,000. This is why compacting the transcript and keeping the context tight is a direct cost control, not just a quality one — every token you keep is paid for again on every step that follows.

3The price surface: model, ratio, caching, batch

With the multiplier understood, now look at the rate it multiplies — because the rate card has more structure than a single number, and that structure is where the cheap wins are. The goal here is to map the four dimensions of the price surface so you know which lever moves your specific bill.

Published rates move, so confirm before you build, but the shape is stable. Using Anthropic's Claude tiers as the worked example (current as of mid-2026): [VERIFY]

Four things to read off this surface. Output costs ~5x input across the tiers — but since agents read far more than they write, your bill still skews to input. The tier spread is 5x from Haiku to Opus on input — the single biggest rate lever you have. Prompt caching lets you store repeated context (system prompt, tool defs, stable documents) and re-read it at a fraction of the input rate — up to roughly a 90% saving on the cached portion. Batch processing halves the rate for work that can tolerate minutes-to-hours of latency. These last two often dwarf the savings from rewriting prompts. [VERIFY: all rates, caching, and batch discounts against current Anthropic pricing before publishing.]

4The biggest lever: route by difficulty

Caching attacks repeated context; this section attacks the other half of the bill — which model handles each step. It is the lever teams most often leave on the table, because the default is so easy to get wrong: pick one capable model and send everything to it.

Here is the waste. Most steps in a real agent workflow are not hard. Classifying an intent, extracting a field, deciding which tool to call, summarizing a result — these are routine, and a small cheap model does them well. Only the genuinely hard steps — complex reasoning, judgment under real ambiguity, long-horizon planning — need a frontier model. Sending everything to the frontier model means paying the top rate for the easy majority. Since the tier spread is roughly 5x (and up to ~25x against the cheapest legacy tiers), routing by difficulty is the highest-leverage rate decision you can make.

The router itself is cheap: deciding "is this step easy or hard?" is a classification task, which is exactly a cheap-model job. You spend a small amount on routing to avoid the premium on the easy majority. The discipline is to make "which model?" a per-step decision, not a one-time default — and to confirm with evaluation that the cheap model actually holds quality on the steps you route to it, rather than assuming it does.

5The other budget: latency vs. accuracy

Cost is tokens; this section is about the other resource an agent spends — time — because the two are linked, and the link is a tradeoff you must set deliberately rather than discover in production. The objective is to make the latency-accuracy curve explicit so you can choose a point on it.

The tension is fundamental. A single model call is fast — often under a second — but on genuinely hard tasks its accuracy plateaus; one shot can only reason so far. Multi-step approaches — planning, tool use, reflection, multi-agent review — push accuracy much higher, but each added step adds a model call, and a deep agentic flow can take many seconds to tens of seconds. You buy accuracy with latency and tokens. There is no free version: more thinking is more correct and more expensive and slower, all at once.

This is why a thinking budget matters: a cap on how many steps, how much reasoning, or how many tokens a task may spend before it must return or escalate. A real-time customer-facing agent has a tight latency budget and should prefer fewer steps and a faster model; an overnight analysis job can spend deeply because no one is waiting. The mistake is having no budget at all — letting the agent reason without limit, which produces the worst case on both axes: slow and expensive, often without a matching accuracy gain. Set the budget from the use case, not the model's appetite.

6The other cost center: infrastructure vs. tokens

Everything so far has been about tokens. But the token invoice is only one of two major cost centers, and a budget that accounts for just one will be wrong. This section is about the other — orchestration, retrieval, observability, retries, and engineering — and, more importantly, about which of the two dominates, because the honest answer is "it depends on your volume," and that dependency is the part most write-ups get wrong.

The non-token costs are real and easy to forget:

1. Orchestration and compute. The servers, queues, and runtime that host the agent and coordinate its steps — especially in multi-agent systems with their coordination overhead.
2. Retrieval infrastructure. Vector databases, embedding generation, and the retrieval pipeline that feeds the agent context — a standing cost that scales with your corpus, not your traffic.
3. Observability and tracing. Non-deterministic systems are undebuggable without tracing, evaluation, and monitoring tooling — a real line item, and not optional.
4. Retries, failures, and remediation. Every retry is more tokens and more compute; every failure that reaches a user is support and trust cost; every incident is engineering time.
5. Engineering and maintenance. Prompt tuning, eval upkeep, security review, and the ongoing work of keeping a probabilistic system behaving — often the largest and least-visible line of all.

Now the part that matters, and the part I want to be precise about rather than wave away: which cost center dominates is not fixed — it flips with your volume. Most of those infrastructure costs are roughly fixed: a vector database, an observability stack, and the engineers maintaining the system cost about the same whether you serve a hundred tasks a day or a million. Token cost, by contrast, is variable — it scales close to linearly with how many tasks you run, amplified by the way each step re-ingests the whole growing context.

This resolves a tension that runs through the whole cost story. Earlier the argument was that tokens explode as the loop re-reads its own context; here the argument is that infrastructure can be the bigger line. Both are true — at different volumes. A low-traffic internal tool is dominated by its fixed stack, and there optimizing tokens is rearranging the small line. A high-traffic customer-facing agent is dominated by tokens, and there the caching and routing levers are the whole game. The mistake is assuming one shape is universal — chasing token savings on a low-volume agent whose real cost is the vector database and the on-call engineer, or ignoring runaway token growth on a high-volume agent because someone read that "most cost is off the invoice."

7The cost-control checklist

The mechanics are complete; this turns them into a build gate. Pull these levers roughly in order — the early ones move the bill most for the least effort.

Conclusion: optimize the multiplier, not the rate

The instinct, when an agent's bill arrives, is to look at the per-token price and shop for a cheaper one. That instinct optimizes the one factor that was already falling and the one that matters least. An agent's cost is the rate times a multiplier — context per step, steps per task, runs per period — and the multiplier is where the bill is made and where it can be unmade.

So attack the multiplier. The loop re-reads its own context on every step, so cache that context and compact the transcript. Most steps are easy, so route them to a cheap model and reserve the frontier for what earns it. More thinking buys accuracy at the price of latency and tokens, so set a budget and stop at it. And remember the token invoice is only one of two cost centers — fixed infrastructure is the other, and which one dominates flips with your volume, so measure your own ratio instead of trusting a headline split.

Measure cost per outcome, not per token; pull caching and routing before you touch anything else; and put a circuit breaker on every run so a single task can never run away. The price per token is the number everyone watches. The multiplier is the number that decides whether your agent is a business or a budget casualty.