# Controlling LLM API Cost: A Practical FinOps Guide

**Scope:** token accounting, model selection, prompt size, retries, caching, and agent limits. This report uses formulas and hypothetical examples; it does not claim industry-wide savings or present unattributed incident stories as fact.

## 1. A useful cost model

For a token-priced request, estimate cost from the actual usage categories in the provider's response:

\[ C = I_u r_{i,u} + I_c r_{i,c} + O r_o + C_{tools} + C_{other} \]

Here, $I_u$ and $I_c$ are uncached and cached input tokens, $O$ is billed output tokens, and the rates are the selected model's current rates. Some services also charge for tools, images, audio, storage, or other units. Do not assume cached input is free or that caching discounts output tokens.

Record usage by model, feature, tenant, and request outcome. Include retries and failed requests. Compare cost per successful task alongside quality and latency; request count alone hides context and output growth.

## 2. Route by measured task quality

Smaller or less expensive models can work well for bounded tasks such as classification, extraction, and short summaries, but performance is task- and data-dependent. Build a representative labeled evaluation set, define acceptable error rates, compare candidate models, and route only where the quality tradeoff is acceptable. Re-evaluate after model or prompt changes.

Use deterministic code when it fully solves the task: schema validation, arithmetic, lookups, and parsing should not incur a generative call by default. Escalation can use confidence or validation signals, but those signals need calibration against held-out examples.

Provider rates and model names change. Consult the [OpenAI pricing page](https://developers.openai.com/api/docs/pricing), [Anthropic pricing](https://docs.anthropic.com/en/docs/about-claude/pricing), and [Vertex AI pricing](https://cloud.google.com/vertex-ai/generative-ai/pricing) when preparing a budget. State the date, model IDs, region, service tier, and token mix in any comparison.

## 3. Context length and retrieval

Every additional input token can add processing cost and latency according to the chosen service's billing and serving behavior. Long-context models may also use information unevenly: Liu et al. documented positional performance changes on specific long-context retrieval tasks. That result is evidence to evaluate context placement, not proof that every long prompt fails. See [Lost in the Middle](https://arxiv.org/abs/2307.03172).

Compare full-context input with retrieval on the same task set. Measure answer quality, citation support, omission rate, end-to-end latency, and total cost, including embeddings, retrieval infrastructure, reranking, and prompt caching. RAG can reduce prompt size, but retrieval can omit relevant evidence; full-context processing can be suitable for one-off synthesis or small corpora.

## 4. Repeated context and caching

Provider prompt caching may reduce the billed cost or processing time for eligible repeated prefixes. Eligibility, minimum prefix size, cache lifetime, and cached-token rates are model-specific; keep these details tied to current documentation. Put stable, reusable instructions and tool definitions before request-specific content when that matches the provider's cache rules. Verify cache usage in response metadata rather than assuming a hit.

Application-level response caching is a separate choice. Exact-match caches are safer for deterministic, non-personalized outputs. Semantic caches can return an answer to a merely similar query; use them only when the answer is valid for the new request, and scope entries by tenant, permissions, locale, model/prompt version, and knowledge-base version as appropriate.

## 5. Agent loops and retries

If a workflow resends growing history on every turn, cumulative input can grow quadratically with the number of turns. Under a simple model with fixed system context $P$, $N$ calls, and $L$ new history tokens added after each call:

\[ I_{total} = NP + L\frac{N(N-1)}{2} \]

This is an illustrative history model, not a universal agent-complexity law. Summarization, truncation, caching, tool payload size, and different turn structures change the total. Bound tool calls and iterations; set request deadlines and output limits; detect repeated actions; and log per-step usage. Retry only errors that are likely to be transient, and cap attempts.

The Python sketch below illustrates a budget guard. Production code should account for provider-specific pricing, concurrent requests, retries, tool charges, and the cost of actions that cannot be rolled back.

```python
class BudgetGuard:
    def __init__(self, max_steps: int, max_cost: float):
        self.max_steps = max_steps
        self.max_cost = max_cost
        self.steps = 0
        self.cost = 0.0

    def record(self, request_cost: float) -> None:
        self.steps += 1
        self.cost += request_cost
        if self.steps > self.max_steps or self.cost > self.max_cost:
            raise RuntimeError("Workflow budget exceeded")
```

## 6. Operational checklist

1. Attribute token and tool usage to a product feature and tenant.
2. Put per-request, per-user, and global budget ceilings in the gateway or orchestration layer.
3. Set input-size, output-size, time, and iteration bounds that fit the product's needs.
4. Alert on unexpected usage rates and compare bills with internal telemetry.
5. Test fallback behavior so budget limits do not silently trigger unsafe or misleading actions.
6. Re-run quality, latency, and cost evaluations before changing model, prompt, retrieval, or cache behavior.

## References

- OWASP GenAI Security Project, [LLM10:2025 Unbounded Consumption](https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/).
- OpenAI, [API pricing](https://developers.openai.com/api/docs/pricing) and [prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching).
- Anthropic, [Pricing](https://docs.anthropic.com/en/docs/about-claude/pricing) and [prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching).
- Google Cloud, [Vertex AI generative AI pricing](https://cloud.google.com/vertex-ai/generative-ai/pricing).
- Liu et al., [Lost in the Middle](https://arxiv.org/abs/2307.03172).