179 questions
No questions match those filters.
Design a production serving stack for a 7B chat model t...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansTo design a production serving stack for a 7B model at this scale, one must first clarify the operational context, specifically whether the workload is interactive or batch-oriented and if it is single or multi-tenant. The architecture should treat prefill and decode phases as distinct components, as they face different computational bottlenecks.
Key steps include estimating the KV cache memory footprint based on the target batch size and identifying the specific constraints that limit throughput. Continuous batching is essential to maintain high utilization and meet the p95 TTFT requirements. Optimization strategies should be discussed only after establishing these fundamental architectural components.