Cross-encoders are more accurate. Why not use one for r...

The primary barrier to using cross-encoders for retrieval is the arithmetic of computational cost. A cross-encoder requires 2P (Lq + Lp) FLOPs per pair, which, when applied to a large corpus (e.g., 21 million passages), results in a retrieval time of roughly 36 minutes per query. This is incompatible with standard production latency budgets, which are often around 50 milliseconds. In contrast, bi-encoders allow the passage tower to be run offline, enabling fast, query-independent retrieval.