Product wants one unified relevance score across all mo...

The conflict arises from conflating a shared numeric scale with a coherent user experience. You should propose per-slot composition: use calibrated per-modality thresholds to decide which items are admitted into the pool, then use a separate, modality-aware presentation layer to order the results. This mirrors how search engines interleave different content types (e.g., maps, videos, links) without forcing them onto a single, mathematically incompatible score.