Product wants to replace the CLIP retrieval backbone wi...

Rather than an all-or-nothing backbone swap, you should implement a funnel pattern. Use the efficient CLIP-based retrieval for the majority of queries to maintain latency SLOs. At query-parsing time, implement a detection mechanism to identify queries that are structurally compositional—those involving multiple entities and explicit spatial relations. Only these specific, complex queries should be routed to the heavier vision-language reranker, ensuring that the latency impact is limited to the subset of traffic that actually requires the higher-capacity model.