← All questions
Inference & ServingVery Hard· Latency· Sampling· SLA

The latency budget dropped from 3s to 1.2s. Eight paral...

Self-probing is often insufficient because it returns a verbalized number with limited support, making it impossible to set precise cutoffs. Instead of switching methods, reduce m to 4. At m=4, the p99 latency is significantly reduced, and the score retains five levels of resolution, which is sufficient for most use cases. You should commit to measuring this by holding coverage fixed at 35% and comparing precision on a labeled split. If this fails to meet the SLA, the p99 target itself should be renegotiated.

Code playground (Python)