← All questions
Retrieval & EmbeddingsMedium· Retrieval· Cosine Similarity· Chunking

What goes wrong when indexing sentences, paragraphs, an...

The pooled cosine of an n-sentence chunk converges to b/√ρ rather than zero. Beyond a specific crossover point, longer chunks outrank shorter, more accurate ones, effectively biasing the results toward length. The fix is to avoid comparing different granularity levels on a raw scale, as the ‘dilution’ effect is often misunderstood as simple noise.

Code playground (Python)