architecture/October 9, 2026

Caching for LLM Applications: Response Reuse and Prompt Prefixes

Three distinct caching mechanisms for LLMs: exact-response caches, semantic cache risks, provider prompt prefix reuse, and cost models.

10 min read

Caching for LLM Applications: Response Reuse and Prompt Prefixes

Purpose: explain three distinct ways to avoid repeated work. Cache hit rates, thresholds, prices, and latency depend on the workload and provider; this report does not assign universal production values.

1. The three mechanisms

Mechanism	Reuses	Application responsibility	Main risk
Exact response cache	A previously generated answer for the same request	Define a complete request key, permissions, expiry, and invalidation	Serving stale or cross-user data
Semantic response cache	An answer for a query judged similar to a new query	Calibrate similarity and verify answer applicability	A near-match can have a different answer
Provider prompt cache	Model-computed state for a matching prompt prefix	Follow model-specific prefix, length, and retention rules; inspect usage	A cache miss or unexpected cache-write charge

These layers are not interchangeable. Response caching can skip generation entirely. Provider prompt caching reuses input processing but still generates a new answer and bills the uncached request content and output.

2. Exact response caching

Create a key from a stable serialization of every input that can change the answer, including where relevant: tenant and authorization scope, model/deployment, system and developer prompts, tool configuration, generation parameters, locale, user input, and the version of retrieved data. Hash the serialized payload if a compact key is useful.

Do not normalize natural-language text in ways that can change meaning. For example, lowercasing may be unsafe when capitalization, identifiers, code, or names matter. Cache only responses that can safely be reused for the full lifetime of the entry. Personal, permission-sensitive, time-sensitive, or action-taking responses often need stricter exclusions or very short expiry.

3. Semantic response caching

A vector similarity score is a retrieval signal, not a guarantee that two questions have the same answer. “Can I return this?” and “Can I return this after 30 days?” may be close in embedding space while requiring different answers. User identity, jurisdiction, product version, dates, and conversation state can also matter.

There is no universal cosine threshold that implies a known false-positive rate. Select a threshold using a labeled validation set representative of real queries. Measure answer-equivalence precision and recall, stratify by sensitive query type, and route uncertain matches to generation. In high-impact settings, avoid semantic answer reuse unless the cached answer is independently grounded and applicability is checked.

Invalidate or version entries when the source documents, access policy, prompt, model, or business rules change. Store provenance with each answer so the system can identify which sources and versions supported it.

4. Provider prompt caching

Provider caching stores reusable model-side state for eligible prompt prefixes. It is not generally a cache of the final answer. The provider still evaluates the request's new content and generates output. Use a stable prefix and put changing session or query content after it when supported by that provider's request format. A cache hit should be confirmed from response usage metadata.

Provider behavior changes by model and API. Minimum prefix length, cache lifetime, read/write pricing, retention controls, and supported inputs are not consistent across vendors or model versions. Check current documentation before deployment:

OpenAI prompt caching and pricing.
Anthropic prompt caching and pricing.
Google Cloud context caching and pricing.

Avoid a cross-provider comparison table with one discount or minimum length per vendor: those values depend on the chosen model and may change. For a budget, record the exact model ID, date checked, cache-read and cache-write rates, and observed cached token count.

5. Cost model

Let $V$ be monthly requests, $h_e$ the exact-response hit fraction, and $h_s$ the semantic-response hit fraction among requests that missed the exact cache. Let $C_e$ be exact-cache cost, $C_s$ the semantic-search and embedding cost, and $C_g$ the generation cost for an unresolved request. The simple expected cost is:

C_{total} = V\left[h_e C_e + (1-h_e)\left(C_s + h_s C_s^{hit} + (1-h_s)C_g\right)\right]

Here, $C_s^{hit}$ is the extra validation or lookup cost for a semantic hit; $C_s$ covers the cost of running semantic search for an exact miss. If embedding cost is incurred for every exact miss, include it once in $C_s$ rather than double-counting it. The model omits cache storage, invalidation, engineering, and quality costs; add them for a total-cost comparison.

Provider prefix caching is a separate adjustment to $C_g$ : split eligible input tokens into uncached, cache-write, and cache-read categories using the selected model's published rates. Do not apply a whole-request discount to output tokens or to requests that did not receive a hit.

Hypothetical arithmetic example

Suppose a workload has 1,000,000 requests, an exact response hit fraction of 0.10, semantic hits equal to 0.20 of exact misses, generation cost of $0.01 per unresolved request, and an all-in semantic lookup cost of$ 0.00001 per exact miss. Ignoring storage and quality costs:

Exact hits: 100,000.
Exact misses: 900,000; semantic hits: 180,000; unresolved requests: 720,000.
Approximate cost: $900,000 \times$ 0.00001 + $720,000 \times$ 0.01 = $7,209.
No-response-cache baseline: $10,000 at the assumed generation cost.

This is a calculation using stated assumptions, not an observed hit rate or a promised saving. The real decision should include semantic errors, storage, and operational cost.

6. Measurement checklist

Track exact and semantic hit rates separately; cache age and invalidation reason; cache key scope; semantic match acceptance and correction rate; provider cached input tokens; end-to-end latency; and cost per successful task. Run a shadow evaluation before enabling semantic response reuse, then review a sample of accepted and rejected matches.

References

OpenAI prompt caching.
Anthropic prompt caching.
Google Cloud context caching overview.
GPTCache project for one open-source semantic-cache implementation; its presence is not evidence of a universal hit rate or threshold.

Cite this research note

BibTeX entry for academic citations, literature trackers, and generative research synthesizers:

@article{glyiq2026_caching_llm_applications,
  title = {Caching for LLM Applications: Response Reuse and Prompt Prefixes},
  author = {Rehan, GlyIQ Infrastructure},
  journal = {GlyIQ Infrastructure Research Notes},
  year = {2026},
  url = {https://glyiq.com/research/caching-llm-applications}
}