How would you design a chatbot that needs to search acr...

At a million documents, a flat vector search is both slow and imprecise, so the fix is to split the work into stages instead of doing it all in one retrieval call. Start with routing: some questions don’t need a document search at all and can be answered directly, so classify that first and skip retrieval when possible. For everything else, combine keyword search (BM25) with vector similarity so the system catches both exact terminology matches and semantically similar phrasing that keyword search alone would miss.

Pull a generous candidate set from that hybrid search — a hundred or so — then use a cross-encoder reranker to narrow it down to the handful that actually matter, since a reranker judges relevance far more precisely than the initial retrieval step. Cache the results for queries that repeat often, since a support or FAQ-style workload has a long tail of identical questions that shouldn’t cost a fresh search every time.

A telecom company running this pattern over 1 million support tickets and policy documents cut latency from 2-3 seconds down to around 400ms and pushed retrieval accuracy from roughly 65% to 89% — the reranking stage did more for accuracy than any embedding-model swap would have.