179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Model Serving· Throughput vs Latency· Speculative Decoding· Resource Allocation
How would you serve the same 70B model to both an inter...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansAttempting to use a single serving configuration for both interactive and batch workloads is suboptimal for both. Instead, you should implement separate serving paths. The interactive path should be optimized for latency, utilizing smaller batches and techniques like speculative decoding. The batch pipeline should be optimized for throughput and cost, utilizing larger batch sizes, more aggressive quantization, and relaxed TTFT SLOs.