Context Stuffing

Context stuffing is the anti-pattern of packing as many retrieved chunks as possible into an LLM prompt on the assumption that more context produces better answers. It became tempting as context windows grew — GPT-4 went from 8k to 32k to 128k tokens; Claude from 100k to 200k to 1M tokens; Gemini to 2M tokens — and the retrieval output of a generous top-k no longer had to fit within a tight budget. Some teams responded by retrieving 50-100 chunks and letting the model sort it out. The empirical result is consistent: quality plateaus and often regresses, latency and cost scale linearly with context size, and the answer-grounding signal the model should follow becomes weaker, not stronger.

The 2023 "Lost in the Middle" paper by Liu et al. made the effect rigorous — LLMs reliably attend more strongly to the start and end of long contexts than the middle, so a relevant passage buried in position 37 of 50 is often effectively invisible. Later work (RULER benchmark, NIAH-variants, Anthropic's needle-in-a-haystack studies) confirmed position sensitivity is still real even at 200k+ contexts, though the frontier models of 2026 have narrowed the gap. In parallel, reranking + tight top-k has emerged as the competing pattern: retrieve generously (top-50), rerank aggressively, pass only the top-3 to 7 chunks to the model. This pattern outperforms stuffing on every enterprise RAG benchmark that measures both quality and cost.

For APAC mid-market teams, the practical discipline is **aim for 3-7 high-relevance chunks, not 30 low-relevance ones**. This requires a reranker, a tight top-k, and measurement of both answer quality and grounding fidelity. Instrument grounding explicitly — does the model's cited passage actually appear in the provided context? — because stuffed contexts often produce answers that cite passages the model has merely inferred from pretraining. Cost and latency matter even when they seem affordable: at production scale, the difference between 5 chunks and 30 chunks per query is real money, and the latency difference shows up in user abandonment.

The non-obvious failure mode is **cost and latency blow-up without quality gain**. A team runs 100 retrieved chunks × 4k tokens each through a frontier model at 10 queries/second and learns at invoice time that context size is the dominant cost driver, not model choice. Worse, the bigger context may have degraded the answer. The fix is discipline about what reaches the prompt — retrieval is cheap, inference is expensive, so spend the retrieval budget on precision (reranking, filtering, deduplication) rather than on passing problems to the model.