Retrieval-Augmented Generation and Long-Context Input

A decision guide for choosing what evidence to send to a model. This report avoids universal claims about context limits, retrieval quality, latency, and savings: those depend on the model, task, corpus, and serving setup.

1. Long context and retrieval solve different problems

Sending a large document or corpus directly to a model is simple and can support synthesis across distant sections. It also sends more input and may increase cost, latency, or the chance that relevant details are overlooked. Retrieval can reduce the prompt to a smaller evidence set, but it can omit the needed passage or lose relationships across documents.

Liu et al. found that performance on their tested multi-document question answering and key-value retrieval tasks often depended on where relevant information appeared in the prompt; performance could decline when evidence was in the middle. This is an empirical result for the studied models and tasks, not a fixed accuracy curve for every current model. Evaluate position sensitivity on your own task. Paper

For standard dense self-attention, attention interactions have a sequence-length-squared component. Optimized attention kernels and model architectures can change practical compute and memory costs. Do not infer end-to-end latency from the asymptotic expression alone; benchmark the target endpoint.

2. A retrieval pipeline to evaluate

One common design is:

1. Split and index documents while retaining source, date, access, and version metadata.
2. Retrieve candidates with lexical search, vector search, or both. Lexical retrieval can help with exact identifiers and terms; embeddings can help with paraphrases.
3. Optionally merge rankings, for example with Reciprocal Rank Fusion (RRF).
4. Optionally rerank candidates with a query-document model.
5. Send selected passages with source identifiers and instructions to answer only from supported evidence.
6. Evaluate the answer and cited evidence; retrieval scores alone do not establish correctness.

RRF is one rank-combination method, not a requirement. A typical form is \sum_m 1/(k + r_m(d)), where r_m(d) is document d's rank in list m. Tune k and candidate counts empirically. See Cormack, Clarke, and Buettcher, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods.

Reranking adds latency and may have separate API or infrastructure cost. Measure whether its improvement in answer quality or evidence recall justifies that cost. Do not assume a fixed number of chunks or a universal token target.

3. Compare architectures with an explicit cost model

For a token-priced model, one request's inference cost is approximately:

Use the actual rates and usage fields for the selected model. For a RAG pipeline, add embedding, search, reranking, storage, and operations costs. Include prompt and tool overhead and any retries. Current rates are published on provider pricing pages, such as OpenAI, Anthropic, and Google Cloud.

Hypothetical per-request comparison

Assume, only for illustration, a model rate of 2.50 per million input tokens and 10 per million output tokens, with 100,000 input tokens and 300 output tokens in the full-context design. Its model cost is 0.253 per request. If a retrieval design sends 2,000 input tokens and the same 300 output tokens, its model cost is 0.008 per request. The model-token difference is $0.245 per request before retrieval, indexing, reranking, storage, and engineering costs.

At one million requests, that token-only difference is $245,000 under these assumptions. This is not a forecast or a benchmark. Change the token counts, output length, prices, cache-hit mix, and infrastructure costs to match the deployment. A different model or pricing tier changes the result.

4. How to choose

Test at least a direct long-context baseline and a retrieval baseline against the same representative query set. Include hard cases that require exact identifiers, evidence from multiple sections, recent documents, and permission filtering. Score:

• Answer correctness and completeness.
• Evidence recall and citation support.
• Performance by evidence position and document length.
• End-to-end p50/p95 latency, including retrieval and reranking.
• Full cost per successful task, including indexing and operational overhead.

Use direct context when the source set is small, a task requires global synthesis, or retrieval misses are unacceptable and the model can handle the input. Use retrieval when the corpus is large or changing, only a subset is relevant, citations and access control are required, or repeated full-context input is too costly. Hybrid designs can retrieve evidence and include a compact summary or selected broader context.

References

• Liu et al., Lost in the Middle: How Language Models Use Long Contexts.
• Cormack, Clarke, and Buettcher, Reciprocal Rank Fusion.
• OpenAI API pricing, Anthropic pricing, and Vertex AI pricing.
• Cohere Rerank documentation for one reranking API; current features and prices should be checked before budgeting.