Two replicas are available. How do you route an incomin...

Effective routing in a multi-replica environment prioritizes cache locality. The system computes the prefix hash of the incoming prompt and checks each replica’s prefix hash table for the longest matching cached prefix. Routing to the replica with the longest match is optimal because it avoids a full re-prefill, which is computationally expensive. If no matches are found, the request is routed to the replica with the most free blocks. Round-robin is used only as a last resort when block counts are equal and no prefix matches exist.