179 questions
No questions match those filters.
← All questions
Inference & ServingMedium· Tokenization· KV Cache· Autoregressive Decoding· Serving Scheduler
Walk me through what happens when a user sends a prompt...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansWhen a user sends a prompt, the system first tokenizes the input. The process then moves to the prefill stage, which is compute-bound and involves writing the KV cache to HBM. After prefill, the system enters the autoregressive decode loop, which is memory-bound and requires reading the KV cache at each step. Finally, the output is detokenized. Throughout this process, a serving scheduler manages the batching of concurrent requests to optimize system performance.