A colleague proposes padding every reranked candidate t...

While the instinct to fix the shape for the compiler is correct, padding to the global maximum is inefficient. For a 110M-parameter model, this approach can increase latency by nearly 8x compared to the original batch. A better approach is to bucket documents into a handful of fixed shapes (e.g., 512, 1024, 2048) to balance compiler efficiency with padding waste.