How would you implement prefix caching across replicas,...

To enable cross-replica prefix caching, a fast key/value service is required to track which replica holds specific cached blocks. When a request arrives, the router checks this distributed store. If a hit is found, the router can either redirect the request to the target replica or perform a block migration via RDMA. Migration is only performed if the shared prefix is large enough (e.g., >512 tokens) to ensure that the latency cost of migration is amortized by the savings in TTFT; otherwise, local re-prefill is more efficient.