An embedding lookup for a 50,000-vocabulary model runs...

First, sort tokens by index within each batch so that consecutive GPU threads access nearby rows, which partially amortizes the waste from 128-byte burst sections. Second, quantize the embedding table to INT8 or INT4; this halves or quarters the bytes per row, allowing each 128-byte burst to deliver more embedding elements. Third, cache hot embeddings in L2 by structuring the training loop to repeat frequent-token batches before evicting, taking advantage of Zipfian distributions where a small percentage of tokens account for the majority of accesses.