If pretraining loss is equal for a wide-shallow and a n...

While pretraining loss may be identical, the operational characteristics differ. Deeper models often perform better on downstream tasks. From an inference perspective, narrow-deep models have smaller KV caches per layer, which reduces memory pressure during long-context generation. Conversely, wider models may offer higher throughput if the tensor parallelism strategy effectively saturates the underlying hardware.