179 questions
No questions match those filters.
A user sends a 10,000-token system prompt. Your TTFT is...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansFixing a high Time To First Token (TTFT) for long prompts requires addressing both architectural and scheduling bottlenecks. First, implement chunked prefill, which allows the system to interleave prompt processing with ongoing decode steps, preventing long prompts from blocking the pipeline. Second, leverage prompt caching; if multiple requests share the same system prompt prefix, compute the KV cache once and reuse it across requests to save redundant computation.
Furthermore, ensure that the infrastructure is optimized by verifying that tensor parallelism overhead is not dominating the latency and that the GPU is actually compute-bound. If the GPU is not fully utilized, the system may be suffering from inefficient resource allocation or communication overhead rather than raw compute limitations.