Your reranker keeps scoring the same handful of documents against new queries all day, but nothing about that computation is cacheable. Why not, and what would you change?

In a bidirectional cross-encoder, the query and document are processed together, meaning no document-only artifact exists to store. By switching to a causal decoder reranker, you can structure the sequence so that the document representation is independent of the query, making it eligible for caching. Approaches like quantization or model shrinking do not solve the fundamental lack of cacheability in the cross-encoder architecture.