← All questions
LLMOps & ProductionVery Hard· System Design· Latency· Evaluation

Your p99 budget is 1.2 s. The ML lead wants IRCoT with...

A five-step loop is mathematically incompatible with a 1.2s budget, as it would lead to a significant budget violation. The design should instead gate the loop so that it only runs for complex queries. Furthermore, because learned depth predictors can degrade, the system should enforce a hard wall-clock limit, emitting a partial answer if the timeout is reached, rather than relying on a fixed step count.

Code playground (Python)