Your multimodal retriever keeps returning the wrong top...

The issue stems from the contrastive objective used to train CLIP, which aligns the entire image with the entire caption. Because the loss function does not distinguish between individual entities within the image or text, the model develops a bias toward the first-mentioned entity. To diagnose this, you should create an entity-swapped probe set—where the order of entities in the caption is changed—to confirm that the model is indeed relying on positional bias rather than actual semantic understanding.