Your support bot’s corpus is 500 stable knowledge-base...

The redundancy stems from the fact that a retrieved chunk’s key and value vectors are deterministic functions of only that chunk’s own tokens. Because these are recomputed on every fetch, the GPU spend scales linearly with query volume despite the underlying data being stable. A strong solution involves precomputing these caches once and reusing them across requests. Weak answers often suggest response-level caching, which fails due to user paraphrasing, or generic batching, which does not address the underlying redundant compute.