What does a "baseline decoder-only transformer" actuall...

“Decoder-only” means every layer uses causal (backwards-only) attention — a token can only attend to itself and earlier tokens, never later ones. That single constraint is what makes the architecture double as both the training objective and the generation procedure: you train it to predict the next token given everything before it, and at inference you run the exact same forward pass one token at a time. There’s no separate encoder stage and no mismatch between how the model trained and how it’s used.

Encoder-decoder architectures (the original Transformer, T5) split understanding and generation into two stacks, which made sense for translation-style tasks with a fixed input/output structure. Decoder- only won for general-purpose LLMs because it’s simpler to scale — one stack, one training objective, one code path — and simplicity compounds favorably as you throw more parameters and data at it.