For a fixed parameter budget, when do you go wider vs....

Depth adds sequential composition — each layer can build on the representation the previous layer produced, which matters for tasks that need multi-step reasoning. But very deep networks are harder to train stably (even with pre-norm and residual connections, gradient and activation dynamics degrade with depth), and layers can’t be parallelized across each other at inference — you pay their latency serially, one after another.

Width adds capacity within a layer — more parallel attention heads and a wider feed-forward block — which trains more stably and parallelizes better on modern accelerators. But widening a layer has diminishing returns past a point; you can’t buy arbitrary reasoning depth just by making every layer bigger.

In practice, most frontier model architectures land on fairly conservative depth-to-width ratios rather than pushing either extreme, because both directions hit real training-stability and hardware- utilization walls well before they hit a hard theoretical limit.