179 questions
No questions match those filters.
← All questions
Inference & ServingMedium· Latency· Prompt Engineering· KV Cache
Your pipeline prepends five 512-token chunks to every prompt and p50 latency is too high. Where is the time going, and what do you change first?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansA strong answer performs the arithmetic: 2,560 prefill tokens, 2N FLOPs per token, and approximately 287 ms on one A100 at 40% MFU. The primary goal is to keep the interface intact while reducing the computational load. Avoid proposing a larger context window, which exacerbates the resource problem, or embedding injection, which is a training-heavy solution.