179 questions
No questions match those filters.
← All questions
System DesignEasy· API Design· Inference· Networking
Walk me through what happens when a user hits your LLM API endpoint.
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansWhen a user hits an LLM API endpoint, the request first undergoes TLS termination and authentication at the gateway. It then passes to the router, which applies routing policies. The request is processed via prefill on either a dedicated or shared worker, and the resulting tokens are streamed incrementally back through the gateway. Metrics like Time To First Token (TTFT) and Inter-Token Latency (ITL) are measured throughout this process.