When is perplexity insufficient as your only signal, ev...

Perplexity is insufficient when the target capability requires compositional generalization rather than simple next-token prediction, such as multi-step mathematical reasoning or long-horizon planning. A model can minimize perplexity on math corpora by learning surface patterns without actually solving novel problems. For these capabilities, you need targeted held-out task evaluations as periodic diagnostics. Frontier labs often use ‘canary tasks’—small, carefully constructed suites evaluated every N steps—to detect when perplexity progress stops translating to genuine capability progress.