179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Latency· Performance Analysis· Distributed Systems
You went from one shard to eight. Corpus size and QPS a...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe bottleneck is the fan-out effect: the query latency is now determined by the slowest of the eight machines. Statistically, the user-visible p99 is approximately the p99.87 percentile of a single machine (0.99^(1/8) = 0.9987). To fix this, consider consolidating back to the minimum number of shards that fit the data, implementing hedged requests against replicas, or converting the fan-out into a routed lookup if the queries contain a usable key.