Your ANN search stage measured 6 ms in staging on a 10...

At 500 million chunks, the index is roughly 19 times larger than at 10 million, meaning it no longer fits on a single GPU card. The search must now occur in CPU DRAM, introducing significant transfer costs. A strong approach involves calculating the memory requirements first and then optimizing the pipeline by hiding the transfer time behind the query-rewrite phase, rather than attempting to force the index into a GPU or reducing embedding dimensions without evidence.