Legal requires every retrieved chunk to carry a clause citation, while the platform requires uniform 512-token chunks to keep ANN latency predictable. How do you adjudicate?

The requirement for uniform 512-token chunks to maintain predictable ANN latency is based on a misunderstanding of how vector search works. Every chunk, regardless of its original text length, is represented as a single d-dimensional vector in the index. Therefore, the search cost is dictated by the total number of vectors (N), the dimensionality (d), and the specific graph parameters of the index, not the size of the underlying text. Variable chunk lengths do, however, affect prefill latency, which should be managed separately from the retrieval latency concerns.