You migrate from a caption-based index to a CLIP-style...

Migrating to a CLIP-style shared embedding space changes the underlying distribution of your vector data, making a one-size-fits-all thresholding approach ineffective. Different content types, particularly layout-heavy documents, will interact with the shared space differently than simple captions.

To manage this, you should segment your corpus based on content type. By computing specific multipliers for these segments, you can tune your reranking and thresholding logic to be sensitive to the nuances of the CLIP space. Finally, propose a hybrid index design that is sized appropriately against your actual budget and performance requirements.