179 questions
No questions match those filters.
← All questions
Inference & ServingMedium· Latency· Scaling· Inference
How would you reduce TTFT for a system serving 10k conc...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansReducing Time To First Token (TTFT) requires addressing two distinct root causes. If long prompts are slowing down prefill, use disaggregated prefill workers or prompt compression to speed up the initial processing. If large queue depths are slowing down dispatch, increase the number of replicas or implement better scheduling priority for new arrivals to ensure they are processed promptly.