An engineering lead argues against building KV-cache re...

While the lead is correct about the arithmetic regarding single-request latency, they are ignoring the impact on fleet-wide capacity. A significant cut in prefill FLOPs allows a fixed number of GPUs to serve more concurrent requests, effectively lowering the cost per request. However, one must consider the exception of streaming UIs where time-to-first-token is critical, and verify what fraction of traffic actually hits the cacheable subset before proceeding.