When fine-tuning earns its cost

Fine-tuning is not the default lever to reach for when a model's outputs aren't quite right. Prompting and retrieval both let a frozen model incorporate new information or behavior at inference time, with no training run, no GPU budget, and no risk of degrading capabilities the base model already had. Fine-tuning is worth its cost when the need is a durable shift in style, format, or domain-specific behavior that would otherwise have to be re-specified in every prompt -- and even then, parameter-efficient methods like LoRA (training small low-rank adapter matrices while freezing the base model) usually beat full fine-tuning on cost without giving up much quality, since only a small fraction of parameters are actually trained.

The failure mode to watch for is treating fine-tuning as a way to teach a model facts. Facts are exactly what retrieval is for -- cheap to update, auditable, and not baked irreversibly into weights. Fine-tuning on a narrow set of facts tends to produce a model that's memorized specific phrasings rather than genuinely learned the domain, and it can't be updated without another training run when those facts change.