Design a serving system for a 7B chat model that needs...

To design this system, you must first calculate the peak memory requirements by accounting for the model parameters and the KV cache at the required batch size. You need to analyze whether the workload is memory-bound or compute-bound at that specific batch size to determine the optimal hardware utilization.

If a single GPU cannot handle the load, you should propose horizontal scaling or model sharding to distribute the request volume. Additionally, implementing continuous batching is essential to maintain high throughput and meet the p95 latency requirements under the 5,000 QPS load.