The cost, latency, and accuracy triangle

Every serving decision trades between three axes, and improving one usually costs one of the other two. A larger model tends to be more accurate but slower and more expensive per token; batching improves aggregate throughput but does not make any single user's response faster, since it trades per-request latency for overall efficiency. The mistake to avoid is optimizing one axis in isolation -- for example reaching for quantization or a smaller model to cut latency without checking whether the accuracy loss actually matters for the task, or whether the real bottleneck was somewhere else in the pipeline entirely (network round trips, a slow retrieval step) rather than the model itself.

Getting this right starts with measuring the actual bottleneck rather than assuming it. A system that spends most of its latency budget on retrieval or on a chain of sequential tool calls will see little benefit from a faster model, and a system that's compute-bound at high concurrency needs a different fix (batching, a smaller model, caching) than one that's latency-bound for a single user.