← All questions
Retrieval & EmbeddingsVery Hard· High-dimensional space· Nearest neighbor search· Evaluation metrics

A colleague cites Beyer et al. and argues that nearest...

The theoretical concern regarding the ‘meaninglessness’ of nearest neighbor search in high dimensions relies on the assumption that data points are distributed without any inherent cluster structure. However, modern embedding models are contrastively trained to map semantic relationships into specific regions of the vector space, effectively creating the structure that the theorem assumes is absent.

Therefore, your empirical result of 0.82 recall@10 is not a violation of the underlying mathematics. Instead, it serves as empirical evidence that your specific data distribution possesses the necessary structure for semantic retrieval to remain effective despite the high dimensionality of the embedding space.

Code playground (Python)