← All questions
Inference & ServingVery Hard· Generative Retrieval· Vector Databases· GPU Utilization· Cost Modeling

Infra says moving to generative retrieval deletes the v...

The adjudication requires separating the storage concerns from the computational concerns. Infrastructure is correct that generative retrieval significantly reduces the memory footprint, as a 29.3 GB replicated host index is replaced by approximately 440 MB of model weights already resident on the GPU. However, the Retrieval team is correct that this does not eliminate the cost; it merely relocates it. The expense is moved from static storage line items into the training and continuous re-training pipeline required to keep the generative model’s internal knowledge updated as the corpus grows by an order of magnitude.

Code playground (Python)