How does eliminating the KV cache change your serving i...

Removing the KV cache fundamentally simplifies the serving stack. Memory usage becomes static and fully predictable before generation begins, meaning the scheduler no longer needs to manage eviction policies or copy-on-write prefix sharing. However, this shift forces a change in how requests are handled, as the system must now commit to an output length upfront, necessitating a robust length-prediction mechanism to avoid inefficiencies.