179 questions
No questions match those filters.
← All questions
LLMOps & ProductionVery Hard· Latency· System Design· Budgeting
End-to-end budget for a conversational turn is 1 second...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansBy calculating the arithmetic of the budget, you can prove that keeping all operations is impossible. Distilling the rewriter to a 1B model in fp16 significantly reduces the decode cost per token. Dropping the HyDE pseudo-response is the most logical step to meet the 1-second budget, as it is computationally expensive. You can mitigate the loss of HyDE by relying on the reranking stage, and only re-add the operation if A/B testing proves that the recall gains justify the latency penalty.